What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret

2025/03/03 by Yufeng Yuan, Yue Yu, Yuan, Yufeng +6 · 22 citations
Chemical Engineering · #Catalysis and Oxidation Reactions #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2503.01491

openalex publication_date 2025/03/03 · openalex created_date 2025/10/12 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.

Cited by

• Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search
• Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
• MURPHY: Multi-Turn GRPO for Self Correcting Code Generation
• Scheduling Your LLM Reinforcement Learning with Reasoning Trees
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
• RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning
• ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework
• CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment

Related

• Truncated Proximal Policy Optimization
• PPO-HSC: An Exploratory Reinforcement Learning Framework Based on Wide-Area Policy Coverage Optimization
• Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length
• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision