A Survey on Large Language Model Benchmarks

2025/08/21 by Shiwen Ni, Guhong Chen, Ni, Shiwen +26 · 1 voice · 16 citations
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2508.15361

openalex publication_date 2025/08/21 · arxiv published 2025/08/21 · arxiv updated 2025/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In recent years, with the rapid development of the depth and breadth of large language models' capabilities, various corresponding evaluation benchmarks have been emerging in increasing numbers. As a quantitative assessment tool for model performance, benchmarks are not only a core means to measure model capabilities but also a key element in guiding the direction of model development and promoting technological innovation. We systematically review the current status and development of large language model benchmarks for the first time, categorizing 283 representative benchmarks into three categories: general capabilities, domain-specific, and target-specific. General capability benchmarks cover aspects such as core linguistics, knowledge, and reasoning; domain-specific benchmarks focus on fields like natural sciences, humanities and social sciences, and engineering technology; target-specific benchmarks pay attention to risks, reliability, agents, etc. We point out that current benchmarks have problems such as inflated scores caused by data contamination, unfair evaluation due to cultural and linguistic biases, and lack of evaluation on process credibility and dynamic environments, and provide a referable design paradigm for future benchmark innovation.

Cited by

• BenCSSmark: Making the Social Sciences Count in LLM Research
• A systematic assessment of Large Language Models for constructing two-level fractional factorial designs
• Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error
• Large language models for automated PRISMA 2020 adherence checking
• Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions
• Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
• On Generalization in Agentic Tool Calling: CoreThink Agentic Reasoner and MAVEN Dataset
• Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
• Enabling Physical AI through Biological Principles
• Geometric Metrics and LLMs: What They Measure and When They Work
• Mapping Overlaps in Benchmarks through Perplexity in the Wild
• AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports
• The Thin Line Between Comprehension and Persuasion in LLMs
• QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation
• MAVEN: Improving Generalization in Agentic Tool Calling
• How Hard is it to Rig a Benchmark? A Social Choice Analysis of Leaderboard Robustness
• Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels

Discussions

• このサーベイ、雑に褒めるより地図として読む方が好き。283件を general / domain-specific / target-specific に分けた上で、漏洩、文化・言語バイアス、静的評価を穴として置いてる。 https://x.com/rohanpaul_ai/status/1959206231215649277 https://arxiv.org/abs/2508.15361 [bsky, 2 points, 2 comments]

Related

• Evaluating Large Language Models: A Comprehensive Survey
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks
• Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
• Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Models
• Beyond Metrics: A Critical Analysis of the Variability in Large Language Model Evaluation Frameworks