Non-Asymptotic Global Convergence of PPO-Clip

2025/12/18 by Liu, Yin, Dai, Qiming, Zhang, Junyu +1
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2512.16565

Abstract

Reinforcement learning (RL) has gained attention for aligning large language models (LLMs) via reinforcement learning from human feedback (RLHF). The actor-only variants of Proximal Policy Optimization (PPO) are widely applied for their efficiency. These algorithms incorporate a clipping mechanism to improve stability. Besides, a regularization term, such as the reverse KL-divergence or a more general \(f\)-divergence, is introduced to prevent policy drift. Despite their empirical success, a rigorous theoretical understanding of the problem and the algorithm's properties is limited. This paper advances the theoretical foundations of the PPO-Clip algorithm by analyzing a deterministic actor-only PPO algorithm within the general RL setting with \(f\)-divergence regularization under the softmax policy parameterization. We derive a non-uniform Lipschitz smoothness condition and a Łojasiewicz inequality for the considered problem. Based on these, a non-asymptotic linear convergence rate to the globally optimal policy is established for the forward KL-regularizer. Furthermore, stationary convergence and local linear convergence are derived for the reverse KL-regularizer.

Citations

• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
• Aligning Language Models with Preferences through f-divergence Minimization
• Improving alignment of dialogue agents via targeted human judgements
• On Decoding Strategies for Neural Text Generators
• Red Teaming Language Models with Language Models
• PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation
• On the Convergence Rates of Policy Gradient Methods
• Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularization
• On the Global Optimum Convergence of Momentum-based Policy Gradient
• A general sample complexity analysis of vanilla policy gradient
• Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
• On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
• Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes
• A Distributional Approach to Controlled Text Generation
• Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER
• Sample Efficient Reinforcement Learning with REINFORCE
• Variational Policy Gradient Method for Reinforcement Learning with General Utilities
• On the Global Convergence Rates of Softmax Policy Gradient Methods
• Leverage the Average: an Analysis of KL Regularization in RL
• Non-asymptotic Convergence of Adam-type Reinforcement Learning Algorithms under Markovian Sampling
• Global Optimality Guarantees For Policy Gradient Methods
• A Theory of Regularized Markov Decision Processes
• Provably Efficient Maximum Entropy Exploration
• Proximal Policy Optimization Algorithms
• A unified view of entropy-regularized Markov decision processes
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Asynchronous Methods for Deep Reinforcement Learning
• Trust Region Policy Optimization

Cited by

• On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence

Related

• Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective
• PPO in the Fisher-Rao geometry
• An Adaptive Clipping Approach for Proximal Policy Optimization
• Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
• REBEL: Reinforcement Learning via Regressing Relative Rewards