StatEval: A Comprehensive Benchmark for Large Language Models in Statistics

2025/10/10 by Yao Lu, Lu, Yuchen, Run Yang +26
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.09517

openalex publication_date 2025/10/10 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Despite rapid advances in large language models (LLMs), statistical reasoning remains underrepresented in existing LLM benchmarks, which often do not reflect the layered, proof-driven nature of real statistical practice. To address this gap, we introduce StatEval, the first large-scale benchmark for statistical reasoning across curricular and research-level settings. StatEval includes over 100,000 curated problems, with 20,000+ foundational questions spanning undergraduate and graduate curricula and 80,000+ research-level proof tasks extracted from leading statistical journals. To construct StatEval, we develop TRACE (Topology and Reasoning-Aware Context Extractor), a multi-agent pipeline with human-in-the-loop validation that converts unstructured academic texts into self-contained theorem-level reasoning tasks. We also propose an Adaptive Process-Based Scoring Pipeline for complex statistical proofs, enabling fine-grained evaluation beyond final-answer matching. Experiments show that while LLMs perform reasonably on foundational tasks, they struggle with rigorous research-level reasoning. Beyond evaluation, StatEval serves as a resource for improving reasoning, as retrieval-augmented generation and domain-specific alignment consistently enhance performance. Together, these results establish StatEval as both a benchmark and an infrastructure for advancing statistical reasoning in LLMs.

Citations

• Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
• R1-RE: Cross-Domain Relation Extraction with RLVR
• EvalAssist: A Human-Centered Tool for LLM-as-a-Judge
• DSCodeBench: A Realistic Benchmark for Data Science Code Generation
• MIRB: Mathematical Information Retrieval Benchmark
• An Empirical Study of Position Bias in Modern Information Retrieval
• RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics
• FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
• AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset
• PaperBench: Evaluating AI's Ability to Replicate AI Research
• DAT: Dynamic Alpha Tuning for Hybrid Retrieval in Retrieval-Augmented Generation
• 1.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Training
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis
• Investigating Non-Transitivity in LLM-as-a-Judge
• To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization
• Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
• MinerU: An Open-Source Solution for Precise Document Content Extraction
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
• PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competition
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
• Are Large Language Models Good Statisticians?
• MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark
• SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation
• Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons
• Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis
• Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
• Nougat: Neural Optical Understanding for Academic Documents
• Scientific discovery in the age of artificial intelligence
• LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• TheoremQA: A Theorem-driven Question Answering dataset
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Solving math word problems with process- and outcome-based feedback
• Autoformalization with Large Language Models
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• A Particle Method for Solving Fredholm Equations of the First Kind
• Measuring Massive Multitask Language Understanding
• Generative Language Modeling for Automated Theorem Proving
• Language Models are Few-Shot Learners
• Bounds on the conditional and average treatment effect with unobserved confounding factors
• Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

Related

• Are Large Language Models Good Statisticians?
• Are Your LLMs Capable of Stable Reasoning?
• Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models
• Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data
• NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes