To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

2021/07/22 by Tom Kocmi, Kocmi, Tom, Christian Federmann +9 · 18 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Software Engineering Research #Text Readability and Simplification #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2107.10821

Accepted to WMT 2021 research papers

openalex publication_date 2021/07/22 · arxiv created 2021/09/13 · arxiv updated 2021/09/15 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on -- to the best of our knowledge -- the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.

Citations

• On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
• A Structured Review of the Validity of BLEU
• Moving to a World Beyond “ p < 0.05”

Cited by

• Hacking Neural Evaluation Metrics with Single Hub Text
• Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domains
• MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
• Sign Language Translation with Sentence Embedding Supervision
• PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions
• Beyond statistical significance: Quantifying uncertainty and statistical variability in multilingual and multitask NLP evaluation
• Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translation
• Summary of ChatGPT-Related research and perspective towards the future of large language models
• Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation
• Findings of the Third Automatic Minuting (AutoMin) Challenge
• Preliminary Ranking of WMT25 General Machine Translation Systems
• Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
• Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
• CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation
• Don't Go Far Off: An Empirical Study on Neural Poetry Translation
• Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models
• TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
• Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling

Related

• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine\n Translation Evaluation Metrics
• A Measure of the System Dependence of Automated Metrics
• Evaluating Automatic Metrics with Incremental Machine Translation Systems
• Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
• Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics