MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

2023/12/28 by Zhongshen Zeng, Pengguang Chen, Zeng, Zhongshen +6 · 10 citations
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.17080

openalex publication_date 2023/12/28 · openalex created_date 2023/12/30 · openalex updated_date 2026/07/28

Abstract

In this work, we introduce a novel evaluation paradigm for Large Language Models (LLMs) that compels them to transition from a traditional question-answering role, akin to a student, to a solution-scoring role, akin to a teacher. This paradigm, focusing on "reasoning about reasoning," hence termed meta-reasoning, shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation that effectively distinguishes between the cognitive capabilities of different models. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark. Our extensive analysis includes several state-of-the-art models from both open-source and commercial domains, uncovering fundamental deficiencies in their training and evaluation methodologies. Notably, while models like Deepseek-v2 and Claude3-Sonnet closely competed with GPT-4 in GSM8K, their performance disparities expanded dramatically in MR-GSM8K, with differences widening to over 20 absolute points, underscoring the significant challenge posed by our meta-reasoning approach.

Cited by

• Accounting Reasoning in Large Language Models: Concepts, Evaluation, and Empirical Analysis
• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
• Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
• PADME: Procedure Aware DynaMic Execution
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
• Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• LLMs cannot spot math errors, even when allowed to peek into the solution
• Can You Trick the Grader? Adversarial Persuasion of LLM Judges

Related

• Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
• Meta Reasoning for Large Language Models
• Are Your LLMs Capable of Stable Reasoning?
• The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
• Generative Evaluation of Complex Reasoning in Large Language Models