On the optimization dynamics of RLVR: Gradient gap and step size thresholds

2025/10/09 by Jin Suk, Suk, Joe, Yaqi Duan +1
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Optimization and Control (math.OC) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2510.08539

openalex publication_date 2025/10/09 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), which uses simple binary feedback to post-train large language models, has found significant empirical success. However, a principled understanding of why it works is lacking. This paper builds a theoretical foundation for RLVR by analyzing its training process at both the full-response (trajectory) and token levels. Central to our analysis is a new quantity called the Gradient Gap, which formalizes the direction of improvement from low-reward to high-reward regions of the response space. We prove that convergence critically depends on aligning the update direction with this Gradient Gap. Moreover, we derive a sharp step-size threshold based on the magnitude of the Gradient Gap: below it, learning converges, whereas above it, performance collapses. Our theory further predicts how the critical step size must scale with response length and the success rate, thereby explaining why practical heuristics such as length normalization improve stability and showing that, with a fixed learning rate, the success rate can stagnate strictly below 100%. Importantly, our theory holds flexibly for any policy-gradient algorithm and so characterizes the dynamics of popular approaches such as REINFORCE and GRPO. We validate these predictions through controlled bandit simulations and language model experiments on post-training Qwen2.5-Math-7B with GRPO.

Citations

• Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
• The Coverage Principle: How Pre-Training Enables Post-Training
• Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
• Outcome-based Exploration for LLM Reasoning
• Group Sequence Policy Optimization
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
• SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression
• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
• Self-Evolving Curriculum for LLM Reasoning
• Understanding R1-Zero-Like Training: A Critical Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration
• Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
• PILAF: Optimal Human Preference Sampling for Reward Modeling
• Process Reinforcement through Implicit Rewards
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Offline Reinforcement Learning for LLM Multi-Step Reasoning
• Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces
• Is RLHF More Difficult than Standard RL?
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
• Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation
• Training language models to follow instructions with human feedback
• On the Convergence Rates of Policy Gradient Methods
• Dueling RL: Reinforcement Learning with Trajectory Preferences
• Training Verifiers to Solve Math Word Problems
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Fine-Tuning Language Models from Human Preferences
• Proximal Policy Optimization Algorithms
• VL Norm: Rethink Loss Aggregation in RLVR

Cited by

• On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence

Related

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• BroRL: Scaling Reinforcement Learning via Broadened Exploration
• Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards