MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models

2024/06/15 by Wentian Wang, Wang, Wentian, Paul B. Kantor +8 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2406.15468

openalex publication_date 2024/06/15 · openalex created_date 2024/06/26 · openalex updated_date 2026/07/28

Abstract

We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that "truly" understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.

Cited by

• Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations

Related

• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
• Are Your LLMs Capable of Stable Reasoning?
• BELL: Benchmarking the Explainability of Large Language Models
• Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models