Stepwise Think-Critique: A Unified Framework for Robust and Interpretable LLM Reasoning

2025/12/17 by Xu, Jiaqi, Lan, Cuiling, Chen, Xuejin +2
Computer Science · #Artificial Intelligence (cs.AI) #Automated reasoning #Case-based reasoning #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Model-based reasoning #Multimodal Machine Learning Applications #Non-monotonic logic #Opportunistic reasoning #Quality (philosophy) #Reasoning system #Topic Modeling

paper · open access · doi:10.48550/arxiv.2512.15662

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/12/17 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Human beings solve complex problems through critical thinking, where reasoning and evaluation are intertwined to converge toward correct solutions. However, most existing large language models (LLMs) treat the reasoning and verification as separate processes: they either generate reasoning without explicit self-checking or rely on external verifiers to detect errors post hoc. The former lacks immediate feedback, while the latter increases system complexity and hinders synchronized learning. Motivated by human critical thinking, we propose Stepwise Think-Critique (STC), a unified and end-to-end trainable framework that interleaves reasoning and self-critique at every intermediate step within a single model. STC is trained with a hybrid reinforcement learning objective that integrates reasoning rewards and critique-consistency rewards, thereby jointly optimizing solution correctness and reliability of self-evaluation. Experiments on mathematical reasoning benchmarks show that STC demonstrates strong critical-thinking capabilities and produces more interpretable reasoning traces, representing a step toward LLMs with built-in critical thinking.

Citations

• StepWiser: Stepwise Generative Judges for Wiser Reasoning
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
• Process Reward Models That Think
• A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• Process Reinforcement through Implicit Rewards
• The Lessons of Developing Process Reward Models in Mathematical Reasoning
• ProcessBench: Identifying Process Errors in Mathematical Reasoning
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Let's Verify Step by Step
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

Related

• Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
• CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
• Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic
• Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
• THiNK: Can Large Language Models Think-aloud?