Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine\n Translation Evaluation Metrics

2020/06/11 by Nitika Mathur, Timothy Baldwin, Mathur, Nitika +3 · 12 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Software Engineering Research #Software System Performance and Reliability #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2006.06264

openalex publication_date 2020/06/11 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

Automatic metrics are fundamental for the development and evaluation of\nmachine translation systems. Judging whether, and to what extent, automatic\nmetrics concur with the gold standard of human evaluation is not a\nstraightforward problem. We show that current methods for judging metrics are\nhighly sensitive to the translations used for assessment, particularly the\npresence of outliers, which often leads to falsely confident conclusions about\na metric's efficacy. Finally, we turn to pairwise system ranking, developing a\nmethod for thresholding performance improvement under an automatic metric\nagainst human judgements, which allows quantification of type I versus type II\nerrors incurred, i.e., insignificant human differences in system quality that\nare accepted, and significant human differences that are rejected. Together,\nthese findings suggest improvements to the protocols for metric evaluation and\nsystem performance evaluation in machine translation.\n

Cited by

• NLG Evaluation: Past, Present, Future
• How to Evaluate Speech Translation with Source-Aware Neural MT Metrics
• A Critical Study of Automatic Evaluation in Sign Language Translation
• MedFusionT5: Cross-Modal Attention Boosts Semantic Quality and Reduces Hallucinations in Dental AI
• SemiAdapt and SemiLoRA: Efficient Domain Adaptation for Transformer-based Low-Resource Language Translation with a Case Study on Irish
• Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
• Don't Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation
• Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation
• The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
• Estimating Machine Translation Difficulty
• TransEvalnia: Reasoning-based Evaluation and Ranking of Translations
• Metric Design != Metric Behavior: Improving Metric Selection for the Unbiased Evaluation of Dimensionality Reduction

Related

• To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for\n Machine Translation
• A Measure of the System Dependence of Automated Metrics
• Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress
• Evaluating Automatic Metrics with Incremental Machine Translation Systems
• Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation