The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

2025/05/28 by Ganqu Cui, Yuchen Zhang, Cui, Ganqu +31 · 159 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.22617

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL runs without entropy intervention, where the policy entropy dropped sharply at the early training stage, this diminished exploratory ability is always accompanied with the saturation of policy performance. In practice, we establish a transformation equation R=-a*e^H+b between entropy H and downstream performance R. This empirical law strongly indicates that, the policy performance is traded from policy entropy, thus bottlenecked by its exhaustion, and the ceiling is fully predictable H=0, R=-a+b. Our finding necessitates entropy management for continuous exploration toward scaling compute for RL. To this end, we investigate entropy dynamics both theoretically and empirically. Our derivation highlights that, the change in policy entropy is driven by the covariance between action probability and the change in logits, which is proportional to its advantage when using Policy Gradient-like algorithms. Empirical study shows that, the values of covariance term and entropy differences matched exactly, supporting the theoretical conclusion. Moreover, the covariance term stays mostly positive throughout training, further explaining why policy entropy would decrease monotonically. Through understanding the mechanism behind entropy dynamics, we motivate to control entropy by restricting the update of high-covariance tokens. Specifically, we propose two simple yet effective techniques, namely Clip-Cov and KL-Cov, which clip and apply KL penalty to tokens with high covariances respectively. Experiments show that these methods encourage exploration, thus helping policy escape entropy collapse and achieve better downstream performance.

Cited by

• Diversity or Precision? A Deep Dive into Next Token Prediction
• DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPO
• A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
• Stable and Efficient Single-Rollout RL for Multimodal Reasoning
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
• ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
• M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization
• GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
• ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
• The Role of Entropy in Visual Grounding: Analysis and Optimization
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
• CARL: Criticality-Aware Agentic Reinforcement Learning
• Rectifying LLM Thought from Lens of Optimization
• Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks
• Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques
• ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
• Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning
• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards
• P1: Mastering Physics Olympiads with Reinforcement Learning
• Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization
• Optimizing Diversity and Quality through Base-Aligned Model Collaboration
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
• Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events?
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
• Towards Understanding Self-play for LLM Reasoning
• Defeating the Training-Inference Mismatch via FP16
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
• Empowering RepoQA-Agent based on Reinforcement Learning Driven by Monte-carlo Tree Search
• Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
• The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation
• Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
• KL-Regularized Reinforcement Learning is Designed to Mode Collapse
• FairGRPO: Fair Reinforcement Learning for Equitable Clinical Reasoning
• GAPO: Robust Advantage Estimation for Real-World Code LLMs
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
• Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
• Soundness-Aware Level: A Microscopic Signature that Predicts LLM Reasoning Potential
• The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
• Demystifying Reinforcement Learning in Agentic Reasoning
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model
• From to : Multidimensional Supervision of Reasoning Process for LLM Optimization
• ExGRPO: Learning to Reason from Experience
• Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
• RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
• Can RL Improve Generalization of LLM Agents? An Empirical Study
• Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models
• DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
• Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning
• Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
• ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping
• LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?
• Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
• PEAR: Phase Entropy Aware Reward for Efficient Reasoning
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection
• The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
• λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
• h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
• Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
• Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information
• Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
• On The Fragility of Benchmark Contamination Detection in Reasoning Models
• Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
• Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts
• Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling
• PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models
• Diversity-Incentivized Exploration for Versatile Reasoning
• Polychromic Objectives for Reinforcement Learning
• Rethinking Entropy Regularization in Large Reasoning Models
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
• STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
• CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• AdaThink-Med: Medical Adaptive Thinking with Uncertainty-Guided Length Calibration
• Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models
• Emergent Slow Thinking in LLMs as Inverse Tree Freezing
• Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning
• Variational Reasoning for Language Models
• Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
• Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
• No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
• Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
• Proximal Supervised Fine-Tuning
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• FlowRL: Matching Reward Distributions for LLM Reasoning
• Single-stream Policy Optimization
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
• Self-Aligned Reward: Towards Effective and Efficient Reasoners
• Towards a Unified View of Large Language Model Post-Training
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• On Entropy Control in LLM-RL Algorithms
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• Jointly Reinforcing Diversity and Quality in Language Model Generations
• Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning
• Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
• Intern-S1: A Scientific Multimodal Foundation Model
• rStar2-Agent: Agentic Reasoning Technical Report
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Semantic Energy: Detecting LLM Hallucination Beyond Entropy
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
• ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism
• CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
• SSRL: Self-Search Reinforcement Learning
• Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
• Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
• Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
• Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoning
• GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following
• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization

Related

• Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
• Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning