Evaluating Step-by-step Reasoning Traces: A Survey

2025/02/17 by Jinu Lee, Julia Hockenmaier, Lee, Jinu +1 · 1 voice · 19 citations
Business, Management and Accounting · Computer Science · #Business Process Modeling and Analysis #Natural Language Processing Techniques #Semantic Web and Ontologies #cs.CL

paper · pdf · doi:10.48550/arxiv.2502.12289

arxiv published 2025/02/17 · arxiv updated 2025/09/20

Abstract

Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research.

Cited by

• Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Proving
• Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error
• Coupled Variational Reinforcement Learning for Language Model General Reasoning
• Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
• An Empirical Study of Reasoning Steps in Thinking Code LLMs
• T-FIX: Text-Based Explanations with Features Interpretable to eXperts
• DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
• The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models
• What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment
• PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs
• From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations
• Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
• CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentation
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning
• Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization
• Gazal-R1: Achieving State-of-the-Art Medical Reasoning with Parameter-Efficient Two-Stage Training
• FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction
• Modeling Beyond MOS: Quality Assessment Models Must Integrate Context, Reasoning, and Multimodality
• Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

Discussions

• arxiv.org/pdf/2502.12289 Evaluating Step-by-step Reasoning Traces: A Survey [bsky, 0 points, 0 comments]

Related

• What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluation
• LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
• Assessing LLM Reasoning Steps via Principal Knowledge Grounding
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
• Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Think