LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods

2025/05/22 by Hyang Cui, Cui, Hyang
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.16129

openalex publication_date 2025/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent studies have applied large language models (LLMs) to machine translation quality estimation (MTQE) by prompting models to assign numeric scores. Nonetheless, these direct scoring methods tend to show low segment-level correlation with human judgments. In this paper, we propose a generation-based evaluation paradigm that leverages decoder-only LLMs to produce high-quality references, followed by semantic similarity scoring using sentence embeddings. We conduct the most extensive evaluation to date in MTQE, covering 8 LLMs and 8 language pairs. Empirical results show that our method outperforms both intra-LLM direct scoring baselines and external non-LLM reference-free metrics from MTME. These findings demonstrate the strength of generation-based evaluation and support a shift toward hybrid approaches that combine fluent generation with accurate semantic assessment.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• The Llama 3 Herd of Models
• The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate
• Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation
• xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection
• OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• COMET: A Neural Framework for MT Evaluation
• BLEURT: Learning Robust Metrics for Text Generation
• BERTScore: Evaluating Text Generation with BERT

Related

• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
• CATER: Leveraging LLM to Pioneer a Multidimensional, Reference-Independent Paradigm in Translation Quality Evaluation
• Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics
• MT-Ranker: Reference-free machine translation evaluation by inter-system ranking
• Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean