Dynamically Allocating Evaluation Effort for Model Ranking

2026/08/04 by Vilém Zouhar, Julia Kreutzer, Alon Lavie +4
Computer Science · #cs.CL #cs.LG

paper · pdf

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Citations

• Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms
• Pearmut: Human Evaluation of Translation Made Trivial
• MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
• MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task
• Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs
• How to Select Datapoints for Efficient Human Evaluation of NLG Models?
• Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons
• Active Evaluation Acquisition for Efficient LLM Benchmarking
• MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task
• K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences
• On Speeding Up Language Model Evaluation
• Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• tinyBenchmarks: evaluating LLMs with fewer examples
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Toward More Effective Human Evaluation for Machine Translation
• Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons
• With Little Power Comes Great Responsibility
• FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms
• Efficient Online Scalar Annotation with Bounded Support
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• A Weighted Correlation Index for Rankings with Ties
• Active Learning with Statistical Models
• Discrete-Variable Extremum Problems