Trust Region Policy Optimization

2015/02/19 by Schulman, John, Levine, Sergey, Moritz, Philipp +2 · 240 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1502.05477

Abstract

We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.

Cited by

• Trust Region Masking for Long-Horizon LLM Reinforcement Learning
• APO: Alpha-Divergence Preference Optimization
• MARPO: A Reflective Policy Optimization for Multi Agent Reinforcement Learning
• Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
• Constrained Reinforcement Learning Using Successor Representations
• Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes
• P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
• Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
• In-Context Learning as Implicit Policy Gradient
• Can an Actor-Critic Optimization Framework Improve Analog Design?
• Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
• Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions
• Performative Policy Gradient: Optimality in Performative Reinforcement Learning
• GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators
• SafeMed-R1: Adversarial Reinforcement Learning for Generalizable and Robust Medical Reasoning in Vision-Language Models
• Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedback
• LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correction
• Non-Asymptotic Global Convergence of PPO-Clip
• Techno-economic optimization of a heat-pipe microreactor, part I: theory and cost optimization
• Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research
• Understanding and Improving Hyperbolic Deep Reinforcement Learning
• Constrained Policy Optimization via Sampling-Based Weight-Space Projection
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Learning Controllable and Diverse Player Behaviors in Multi-Agent Environments
• GPG: Generalized Policy Gradient Theorem for Transformer-based Policies
• SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation
• MOA: Multi-Objective Alignment for Role-Playing Agents
• Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement
• Learning Without Time-Based Embodiment Resets in Soft-Actor Critic
• A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation
• A Fast Anti-Jamming Cognitive Radar Deployment Algorithm Based on Reinforcement Learning
• Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning
• Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges
• Digital Twin-based Control Co-Design of Full Vehicle Active Suspensions via Deep Reinforcement Learning
• Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
• On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
• The Silence that Speaks: Neural Estimation via Communication Gaps
• Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO
• Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
• Reinforcing Action Policies by Prophesying
• Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation
• ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal Diffusion
• Curvature-Aware Safety Restoration In LLMs Fine-Tuning
• Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models
• Stabilizing Policy Gradient Methods via Reward Profiling
• Entropy-Based Measurement of Value Drift and Alignment Work in Large Language Models
• GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• Collaborative Charging Optimization for Wireless Rechargeable Sensor Networks via Heterogeneous Mobile Chargers
• Evaluating Model-Agnostic Meta-Learning on MetaWorld ML10 Benchmark: Fast Adaptation in Robotic Manipulation Tasks
• Reinforcement Learning for Chemical Ordering in Alloy Nanoparticles
• Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy
• Intelligent Collaborative Optimization for Rubber Tyre Film Production Based on Multi-path Differentiated Clipping Proximal Policy Optimization
• Conformal Constrained Policy Optimization for Cost-Effective LLM Agents
• Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks
• Harnessing Bounded-Support Evolution Strategies for Policy Refinement
• Black-Box On-Policy Distillation of Large Language Models
• Data Fusion-Enhanced Decision Transformer for Stable Cross-Domain Generalization
• PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore
• On Geometric Structures for Policy Parameterization in Continuous Control
• Achieving fast and robust perfect entangling gates via reinforcement learning
• Secure Low-altitude Maritime Communications via Intelligent Jamming
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• Policy Gradient-Based EMT-in-the-Loop Learning to Mitigate Sub-Synchronous Control Interactions
• On Flow Matching KL Divergence
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
• RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
• Scaling Agent Learning via Experience Synthesis
• NCSAC: Effective Neural Community Search via Attribute-augmented Conductance
• Tensor-Efficient High-Dimensional Q-learning
• Leveraging Discrete Function Decomposability for Scientific Design
• Directional-Clamp PPO
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
• Reinforcement learning based data assimilation for unknown state model
• Second-Order Policy Gradient Methods for the Linear Quadratic Regulator
• Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning
• KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
• Accelerating Trust-Region Methods: An Attempt to Balance Global and Local Efficiency
• Adaptive Inverse Kinematics Framework for Learning Variable-Length Tool Manipulation in Robotics
• Data-Efficient RLVR via Off-Policy Influence Guidance
• ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems
• Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning
• Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error
• Model-Free Robust Beamforming in Satellite Downlink using Reinforcement Learning
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• A Benchmark Study of Deep Reinforcement Learning Algorithms for the Container Stowage Planning Problem
• Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
• SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space
• Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering
• Latent Chain-of-Thought for Visual Reasoning
• Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• Actor-Critic Learning for Risk-Constrained Linear Quadratic Regulation
• Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
• How Hard is it to Confuse a World Model?
• DEEDEE: Fast and Scalable Out-of-Distribution Dynamics Detection
• ProSh: Probabilistic Shielding for Model-free Reinforcement Learning
• Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach
• Generative Reasoning Recommendation via LLMs
• Continual Knowledge Adaptation for Reinforcement Learning
• A Communication-Efficient Decentralized Actor-Critic Algorithm
• Learning Boltzmann Generators via Constrained Mass Transport
• Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPs
• ADPO: Anchored Direct Preference Optimization
• R2L: Reliable Reinforcement Learning: Guaranteed Return & Reliable Policies in Reinforcement Learning
• Adversarial Reinforcement Learning for Robust Control of Fixed-Wing Aircraft under Model Uncertainty
• STABLE: Gated Continual Learning for Large Language Models
• Convergence of actor-critic for entropy regularised MDPs in general action spaces
• Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion
• AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisions
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• A Task-Efficient Reinforcement Learning Task-Motion Planner for Safe Human-Robot Cooperation
• Bayesian Optimization for Dynamic Pricing and Learning
• Deep SPI: Safe Policy Improvement via World Models
• Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
• Escaping Local Optima in the Waddington Landscape: A Two-Stage TRPO-PPO Approach for Single-Cell Perturbation Analysis
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• ExGRPO: Learning to Reason from Experience
• Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processes
• Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
• Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
• Towards Information-Optimized Multi-Agent Path Finding: A Hybrid Framework with Reduced Inter-Agent Information Sharing
• Evaluation of a Robust Control System in Real-World Cable-Driven Parallel Robots
• SIMU: Selective Influence Machine Unlearning
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Medical Vision Language Models as Policies for Robotic Surgery
• ARM: Discovering Agentic Reasoning Modules for Generalizable Multi-Agent Systems
• Policy Gradient Guidance Enables Test Time Control
• Risk-Sensitive Option Market Making with Arbitrage-Free eSSVI Surfaces: A Constrained RL and Stochastic Control Bridge
• DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
• Tail-Safe Hedging: Explainable Risk-Sensitive Reinforcement Learning with a White-Box CBF--QP Safety Layer in Arbitrage-Free Markets
• A KL-regularization Framework for Learning to Plan with Adaptive Priors
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• Generalized Fitted Q-Iteration with Clustered Data
• FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Management
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• It Takes Two: Your GRPO Is Secretly DPO
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• BroRL: Scaling Reinforcement Learning via Broadened Exploration
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
• Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
• Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
• Polychromic Objectives for Reinforcement Learning
• Sampling Complexity of TD and PPO in RKHS
• Discrete Variational Autoencoding via Policy Search
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Dual-Scale World Models for LLM Agents Towards Hard-Exploration Problems
• Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training
• Integrated Communication and Control for Energy-Efficient UAV Swarms: A Multi-Agent Reinforcement Learning Approach
• Adversarial Diffusion for Robust Reinforcement Learning
• Anchored Supervised Fine-Tuning
• Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization
• ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse
• Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
• Causally-Enhanced Reinforcement Policy Optimization
• Solve Smart, Not Often: Policy Learning for Costly MILP Re-solving
• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
• Rethinking Large Language Model Distillation: A Constrained Markov Decision Process Perspective
• It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
• ImaginationPolicy: Towards Generalizable, Precise and Reliable End-to-End Policy for Robotic Manipulation
• d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
• Learning to Lead Themselves: Agentic AI in MAS using MARL
• Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation
• Future Policy Aware Preference Learning for Mathematical Reasoning
• Policy Gradient Steering: Interventions from Behavioral Objectives
• Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
• Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
• Physical Embodiment Enables Information Processing Beyond Explicit Sensing in Active Matter
• Proximal Supervised Fine-Tuning
• Evaluation-Aware Reinforcement Learning
• Towards Provable Emergence of In-Context Reinforcement Learning
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories
• From Uniform to Heterogeneous: Tailoring Policy Optimization to Every Token's Nature
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Foundation Models as World Models: A Foundational Study in Text-Based GridWorlds
• Constructive Conflict-Driven Multi-Agent Reinforcement Learning for Strategic Diversity
• Co-Alignment: Rethinking Alignment as Bidirectional Human-AI Cognitive Adaptation
• What Matters in Data for DPO?
• Gradient Free Deep Reinforcement Learning With TabPFN
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning
• Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
• Language Self-Play For Data-Free Training
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Bridging the Gap Between Theoretical and Practical Reinforcement Learning in Undergraduate Education
• Greener Deep Reinforcement Learning: Analysis of Energy and Carbon Efficiency Across Atari Benchmarks
• Towards a Unified View of Large Language Model Post-Training
• Reinforcement Learning-based Control via Y-wise Affine Neural Networks (YANNs)
• Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient
• Towards Scalable O-RAN Resource Management: Graph-Augmented Proximal Policy Optimization
• Reinforcement Learning for Machine Learning Engineering Agents
• Sharpe Ratio Optimization in Markov Decision Processes
• Machine Intelligence on the Edge: Interpretable Cardiac Pattern Localisation Using Reinforcement Learning
• Convergence of regularized agent-state-based Q-learning in POMDPs
• SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
• History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
• Breaking Through Barren Plateaus: Reinforcement Learning Initializations for Deep Variational Quantum Circuits
• Language Models For Generalised PDDL Planning: Synthesising Sound and Programmatic Policies
• Adversarial Agent Behavior Learning in Autonomous Driving Using Deep Reinforcement Learning
• In2x at WMT25 Translation Task
• Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference
• Efficient Environment Design for Multi-Robot Navigation via Continuous Control
• Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces
• Object Fidelity Diffusion for Remote Sensing Image Generation
• Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• PCHands: PCA-based Hand Pose Synergy Representation on Manipulators with N-DoF
• Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
• Learning to control inexact Benders decomposition via reinforcement learning
• Reparameterization Proximal Policy Optimization
• Integrating Vision Foundation Models with Reinforcement Learning for Enhanced Object Interaction
• CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL
• Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• D2PPO: Diffusion Policy Policy Optimization with Dispersive Loss
• Hierarchical Learning-Based Control for Multi-Agent Shepherding of Stochastic Autonomous Agents
• Proactive Constrained Policy Optimization with Preemptive Penalty
• Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
• Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
• One-Step Flow Policy Mirror Descent
• Learning to Drift with Individual Wheel Drive: Maneuvering Autonomous Vehicle at the Handling Limits
• Model Predictive Adversarial Imitation Learning for Planning from Observation
• Flow Matching Policy Gradients
• Reinforcement Learning for Multi-Objective Multi-Echelon Supply Chain Optimisation
• Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning
• Policy Disruption in Reinforcement Learning:Adversarial Attack with Large Language Models and Critical State Identification
• Multi-Agent Guided Policy Optimization
• PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
• Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
• The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

Related

• Quasi-Newton Trust Region Policy Optimization
• Learning to Constrain Policy Optimization with Virtual Trust Region
• Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
• Matrix Low-Rank Trust Region Policy Optimization
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization