Evaluating Step-by-step Reasoning Traces: A Survey

2025/02/17 by Julia Hockenmaier, Lee, Jinu, Hockenmaier, Julia · 14 citations
Computer Science · Business, Management and Accounting · #Semantic Web and Ontologies #Business Process Modeling and Analysis #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2502.12289

Abstract

Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research.

Cited by

• Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Proving
• Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error
• Coupled Variational Reinforcement Learning for Language Model General Reasoning
• Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
• An Empirical Study of Reasoning Steps in Thinking Code LLMs
• T-FIX: Text-Based Explanations with Features Interpretable to eXperts
• DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
• The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models
• What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment
• PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs
• From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations
• Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
• CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentation
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning

Related

• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
• LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
• Assessing LLM Reasoning Steps via Principal Knowledge Grounding
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
• Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think