Extending Automatic Machine Translation Evaluation to Book-Length Documents

2025/09/21 by Wang, Kuang-Da, Ding, Shuoyang, Yang, Chao-Han Huck +4 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.17249

Abstract

Despite Large Language Models (LLMs) demonstrating superior translation performance and long-context capabilities, evaluation methodologies remain constrained to sentence-level assessment due to dataset limitations, token number restrictions in metrics, and rigid sentence boundary requirements. We introduce SEGALE, an evaluation scheme that extends existing automatic metrics to long-document translation by treating documents as continuous text and applying sentence segmentation and alignment methods. Our approach enables previously unattainable document-level evaluation, handling translations of arbitrary length generated with document-level prompts while accounting for under-/over-translations and varied sentence boundaries. Experiments show our scheme significantly outperforms existing long-form document evaluation schemes, while being comparable to evaluations performed with groundtruth sentence alignments. Additionally, we apply our scheme to book-length texts and newly demonstrate that many open-weight LLMs fail to effectively translate documents at their reported maximum context lengths.

Citations

• MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
• EuroLLM: Multilingual Language Models for Europe
• Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models
• Improving Long Context Document-Level Machine Translation
• Escaping the sentence-level paradigm in machine translation
• Large language models effectively leverage document-level context for literary translation, but critical errors persist
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature
• Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric
• COMET: A Neural Framework for MT Evaluation
• BLEURT: Learning Robust Metrics for Text Generation
• Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation
• Selective Attention for Context-aware Neural Machine Translation
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Document-Level Neural Machine Translation with Hierarchical Attention Networks
• Representation Learning with Contrastive Predictive Coding
• Evaluating Discourse Phenomena in Neural Machine Translation
• Does Neural Machine Translation Benefit from Larger Context?

Cited by

• Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics

Related

• Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends
• Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models
• Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
• Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation
• Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level