Crosslingual Optimized Metric for Evaluation of Translation (COMET)

Crosslingual Optimized Metric for Evaluation of Translation (COMET) is a metric for automatic evaluation of machine translation that calculates the similarity between a machine translation output and a reference translation using token or sentence embeddings.

 

COMET supports the Robustness objective by providing a reliable and consistent method for evaluating machine translation outputs. By correlating well with human judgments, it helps ensure that translation systems maintain quality and performance across different languages and conditions, which is a key aspect of robustness. However, its connection to other Trustworthy AI objectives is minimal, as it does not directly address transparency, explainability, or other ethical and governance concerns.

Trustworthy AI Relevance

This metric addresses Robustness, Fairness by quantifying relevant system properties. COMET supports Robustness by providing a reliable and consistent measure of translation quality across different languages and conditions, helping to ensure AI systems maintain performance under diverse and potentially adverse linguistic scenarios. It supports Fairness by enabling the detection and mitigation of biases or quality disparities in machine translation outputs across languages, promoting equitable treatment of different language communities.