Are We Done with MMLU?

2024/06/06 by Aryo Pradipta Gema, Joshua Ong Jun Leang, Gema, Aryo Pradipta +29 · 5 voices · 77 citations
Computer Science · #Business #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2406.04127

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

Cited by

• Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
• Athena-Brain Technical Report: An Efficient Robot Brain for General Intelligence and Embodied Interaction
• SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs
• Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs
• xHC: Expanded Hyper-Connections
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Benchmarks Saturate When The Model Gets Smarter Than The Judge
• Kimi Linear: An Expressive, Efficient Attention Architecture
• Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
• REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
• What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks
• Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Sigma-MoE-Tiny Technical Report
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
• Invisible Hands: Gray-Box Bit Flip Attack for Steering LLMs Without Knowledge of Gradients, Data, and Weights
• Solving Spatial Supersensing Without Spatial Supersensing
• LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
• Unsteady Metrics and Benchmarking Cultures of AI Model Builders
• Motif 2 12.7B technical report
• Reusing Pre-Training Data at Test Time is a Compute Multiplier
• DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
• MemSFT: Mitigating Alignment Tax with an External Parametric Memory
• Ministral 3
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
• Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?
• EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
• When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling
• Distractor Injection Attacks on Large Reasoning Models: Characterization and Defense
• Selective Labeling with False Discovery Rate Control
• Cache-to-Cache: Direct Semantic Communication Between Large Language Models
• The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
• RADAR: Reasoning-Ability and Difficulty-Aware Routing for Reasoning LLMs
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
• Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
• Introducing LongCat-Flash-Thinking: A Technical Report
• Towards Understanding Visual Grounding in Visual Language Models
• Merge-of-Thought Distillation
• Universal Properties of Activation Sparsity in Modern Large Language Models
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
• STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports
• OverFill: Two-Stage Models for Efficient Language Model Decoding
• VGGSounder: Audio-Visual Evaluations for Foundation Models
• Test Set Quality in Multilingual LLM Evaluation
• Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
• Kimi K2: Open Agentic Intelligence
• Mellum2 Technical Report
• ERNIE 5.0 Technical Report
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning
• Deprecating Benchmarks: Criteria and Framework
• Training Language Model to Critique for Better Refinement
• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers
• Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
• Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
• PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
• Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition
• Knowledge Grafting of Large Language Models
• CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
• Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
• Let LRMs Break Free from Overthinking via Self-Braking Tuning
• Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models
• Deep Research Pretraining via Predictive Navigation
• LLMs Outperform Experts on Challenging Biology Benchmarks
• Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
• Diffusion Language Models: An Experimental Analysis
• Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
• Improved Large Language Diffusion Models
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
• SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining
• The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
• Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

Discussions

• Our work on fixing and improving MMLU ("Are We Done with MMLU?", arxiv.org/abs/2406.04127; NAACL 2025) is featured on the DeepSeek home page! deepseek.com -- it's always amazing to see real-world appl [bsky, 23 points, 0 comments]
• Super Cool work from Cohere for AI! 🎉 However, this highlights a concern raised by our MMLU-Redux team (arxiv.org/abs/2406.04127): **error propagation to many languages**. Issues in MMLU (e.g., "rapi [bsky, 9 points, 1 comments]
• Very cool work! 👏🚀 Unfortunately, errors in the original dataset will propagate to all new languages 😕 We investigated the issue of existing errors in the original MMLU in arxiv.org/abs/2406.0412 [bsky, 4 points, 0 comments]
• On SWEBench Pro, 68.5% of GPT 5.5’s failures were caused by broken or incorrect test cases, totaling 28.9% of the entire benchmark [lemmy, 1 points, 0 comments]
• "Are We Done with MMLU" made it to #NAACL2025. Massive congrats to the team especially @aryopg.bsky.social. 🚀 📋 Preprint: arxiv.org/abs/2406.04127 👨🏻‍💻 GitHub: github.com/aryopg/mmlu-... 🤗 Hu [bsky, 0 points, 0 comments]

Related

• MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Changing Answer Order Can Decrease MMLU Accuracy
• MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models
• Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM