Rectifying LLM Thought from Lens of Optimization

2025/12/01 by Liu, Junnan, Liu, Hongwei, Zhang, Songyang +1
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2512.01925

Abstract

Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.

Citations

• Group Sequence Policy Optimization
• LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• RLPR: Extrapolating RLVR to General Domains without Verifiers
• Reasoning with Exploration: An Entropy Perspective
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Reinforcing General Reasoning without Verifiers
• ARM: Adaptive Reasoning Model
• Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
• SituatedThinker: Grounding LLM Reasoning with Real-World through Situated Thinking
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
• Thinkless: LLM Learns When to Think
• AdaptThink: Reasoning Models Can Learn When to Think
• Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
• Qwen3 Technical Report
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
• A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning
• Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
• ToRL: Scaling Tool-Integrated RL
• Understanding R1-Zero-Like Training: A Critical Perspective
• Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
• An Empirical Study on Eliciting and Improving R1-like Reasoning Models
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thought
• Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
• Are Your LLMs Capable of Stable Reasoning?
• Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
• Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?
• Evaluation of OpenAI o1: Opportunities and Challenges of AGI
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Large Language Models for Mathematical Reasoning: Progresses and Challenges
• Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
• SGLang: Efficient Execution of Structured Language Model Programs
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Let's Verify Step by Step
• Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
• Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Looped Transformers as Programmable Computers
• Tighter Bounds on the Expressivity of Transformer Encoders
• Towards Reasoning in Large Language Models: A Survey
• Transformers Learn Shortcuts to Automata
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Saturated Transformers are Constant-Depth Threshold Circuits
• Thinking Like Transformers
• Self-Attention Networks Can Process Bounded Hierarchical Languages
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Measuring Mathematical Problem Solving With the MATH Dataset
• RNNs can generate bounded hierarchical languages with optimal memory
• Measuring Massive Multitask Language Understanding
• O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Are Transformers universal approximators of sequence-to-sequence functions?
• Fast Transformer Decoding: One Write-Head is All You Need
• Universal Transformers
• Proximal Policy Optimization Algorithms
• Attention Is All You Need
• Language Modeling with Gated Convolutional Networks
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

Related

• Demystifying Long Chain-of-Thought Reasoning in LLMs
• Direct Reasoning Optimization: LLMs Can Reward And Refine Their Own Reasoning for Open-Ended Tasks
• Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought