High-Dimensional Continuous Control Using Generalized Advantage Estimation

2015/06/08 by Schulman, John, Moritz, Philipp, Levine, Sergey +2 · 237 citations
#FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.1506.02438

Abstract

Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficulty of obtaining stable and steady improvement despite the nonstationarity of the incoming data. We address the first challenge by using value functions to substantially reduce the variance of policy gradient estimates at the cost of some bias, with an exponentially-weighted estimator of the advantage function that is analogous to TD(lambda). We address the second challenge by using trust region optimization procedure for both the policy and the value function, which are represented by neural networks. Our approach yields strong empirical results on highly challenging 3D locomotion tasks, learning running gaits for bipedal and quadrupedal simulated robots, and learning a policy for getting the biped to stand up from starting out lying on the ground. In contrast to a body of prior work that uses hand-crafted policy representations, our neural network policies map directly from raw kinematics to joint torques. Our algorithm is fully model-free, and the amount of simulated experience required for the learning tasks on 3D bipeds corresponds to 1-2 weeks of real time.

Cited by

• MARPO: A Reflective Policy Optimization for Multi Agent Reinforcement Learning
• Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
• On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
• Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
• PLATO: Pointer Learner for Agent and Task Openness
• Reinforcement Learning for Heterogeneous Sensor Selection in Maritime Surveillance
• Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
• RLLaVA: An RL-central Framework for Language and Vision Assistants
• Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight
• Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedback
• Learning to Design City-scale Transit Routes
• LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correction
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
• Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making
• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
• A Scientific Reasoning Model for Organic Synthesis Procedure Generation
• PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations
• HetRL: Efficient Reinforcement Learning for LLMs in Heterogeneous Environments
• Deep Hedging with Reinforcement Learning: A Practical Framework for Option Risk Management
• Three methods, one problem: Classical and AI approaches to no-three-in-line
• Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning
• GPG: Generalized Policy Gradient Theorem for Transformer-based Policies
• HypeR Adaptivity: Joint hr-Adaptive Meshing via Hypergraph Multi-Agent Deep Reinforcement Learning
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering
• Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions
• Quantifying Memory Use in Reinforcement Learning with Temporal Range
• Entropy-Controlled Intrinsic Motivation Reinforcement Learning for Quadruped Robot Locomotion in Complex Terrains
• RLAX: Large-Scale, Distributed Reinforcement Learning for Large Language Models on TPUs
• Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulation
• Auto-exploration for online reinforcement learning
• A Fast Anti-Jamming Cognitive Radar Deployment Algorithm Based on Reinforcement Learning
• STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Models
• DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
• Autonomous Reinforcement Learning Robot Control with Intel's Loihi 2 Neuromorphic Hardware
• Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GA
• Crossing the Sim2Real Gap Between Simulation and Ground Testing to Space Deployment of Autonomous Free-flyer Control
• SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
• From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
• AID: Agent Intent from Diffusion for Multi-Agent Informative Path Planning
• Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
• Rectifying LLM Thought from Lens of Optimization
• High entropy leads to symmetry-equivariant policies in Dec-POMDPs
• On the Tension Between Optimality and Adversarial Robustness in Policy Optimization
• Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
• Safe and Sustainable Electric Bus Charging Scheduling with Constrained Hierarchical DRL
• SpeedAug: Policy Acceleration via Tempo-Enriched Policy and RL Fine-Tuning
• Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
• Learning Branching Policies for MILPs with Proximal Policy Optimization
• Deep Gaussian Process Proximal Policy Optimization
• Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
• The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality
• Masked-and-Reordered Self-Supervision for Reinforcement Learning from Verifiable Rewards
• Multi-Agent Pointer Transformer: Seq-to-Seq Reinforcement Learning for Multi-Vehicle Dynamic Pickup-Delivery Problems
• Human Imitated Bipedal Locomotion with Frequency Based Gait Generator Network
• Object-Centric World Models for Causality-Aware Reinforcement Learning
• Transformer-Based Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions
• Reinforcement Learning for Chemical Ordering in Alloy Nanoparticles
• HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning
• Reinforcing Trustworthiness in Multimodal Emotional Support Systems
• DemoTuner: Efficient DBMS Knobs Tuning via LLM-Assisted Demonstration Reinforcement Learning
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
• CoRL-MPPI: Enhancing MPPI With Learnable Behaviours For Efficient And Provably-Safe Multi-Robot Collision Avoidance
• Toward Honest Language Models for Deductive Reasoning
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
• PrefPoE: Advantage-Guided Preference Fusion for Learning Where to Explore
• Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
• LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration
• Superhuman AI for Stratego Using Self-Play Reinforcement Learning and Test-Time Search
• Deep Reinforcement Learning for Dynamic Origin-Destination Matrix Estimation in Microscopic Traffic Simulations Considering Credit Assignment
• Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning
• Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
• Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning
• Reinforcement learning based data assimilation for unknown state model
• A Quantitative Comparison of Centralised and Distributed Reinforcement Learning-Based Control for Soft Robotic Arms
• KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
• GraphChain: Large Language Models for Large-scale Graph Analysis via Tool Chaining
• π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
• ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents
• When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making
• Neural Co-state Policies: Structuring Hidden States in Recurrent Reinforcement Learning
• Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning
• Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering
• ZTRS: Zero-Imitation End-to-end Autonomous Driving with Trajectory Scoring
• Sequential Multi-Agent Dynamic Algorithm Configuration
• Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach
• Rethinking GSPO: The Perplexity-Entropy Equivalence
• Offline Preference Optimization via Maximum Marginal Likelihood Estimation
• FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning
• Reinforcement learning-guided optimization of critical current in high-temperature superconductors
• Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
• SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph
• Extracting alignment data in open models
• Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
• Agentic Reinforcement Learning for Search Misaligns Instruction-Tuning
• Fine-tuning Flow Matching Generative Models with Intermediate Feedback
• Plasma Shape Control via Zero-shot Generative Reinforcement Learning
• SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
• Procedural Game Level Design with Deep Reinforcement Learning
• RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning
• Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion
• Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigm
• ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
• Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
• Self-Verifying Reflection Helps Transformers with CoT Reasoning
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learning
• Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
• CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation
• Towards Dynamic Quadrupedal Gaits: A Symmetry-Guided RL Hierarchy Enables Free Gait Transitions at Varying Speeds
• Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models
• Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning
• DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
• Towards Information-Optimized Multi-Agent Path Finding: A Hybrid Framework with Reduced Inter-Agent Information Sharing
• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation
• RLinf-VLA: A Unified and Efficient Framework for VLA+RL Training
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Reference Grounded Skill Discovery
• Information-Theoretic Policy Pre-Training with Empowerment
• Policy Gradient Guidance Enables Test Time Control
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• Risk-Sensitive Option Market Making with Arbitrage-Free eSSVI Surfaces: A Constrained RL and Stochastic Control Bridge
• Tail-Safe Hedging: Explainable Risk-Sensitive Reinforcement Learning with a White-Box CBF--QP Safety Layer in Arbitrage-Free Markets
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework
• TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
• Group Policy Gradient
• FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Management
• GEM: A Gym for Agentic LLMs
• It Takes Two: Your GRPO Is Secretly DPO
• Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
• Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
• VLA Model Post-Training via Action-Chunked PPO and Self Behavior Cloning
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
• Polychromic Objectives for Reinforcement Learning
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
• Quantifying Generalisation in Imitation Learning
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables
• GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
• Adversarial Reinforcement Learning Framework for ESP Cheater Simulation
• Multi-Agent Guided Policy Search for Non-Cooperative Dynamic Games
• GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoning
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
• Integrated Communication and Control for Energy-Efficient UAV Swarms: A Multi-Agent Reinforcement Learning Approach
• Beyond the Exploration-Exploitation Trade-off: A Hidden State Approach for LLM Reasoning in RLVR
• CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning
• GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
• Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
• Tree Search for LLM Agent Reinforcement Learning
• Evading Overlapping Community Detection via Proxy Node Injection
• CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning
• Complexity-Regularized Proximal Policy Optimization
• Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
• Learning to Lead Themselves: Agentic AI in MAS using MARL
• An effective control of large systems of active particles: An application to evacuation problem
• Robot Trajectron V2: A Probabilistic Shared Control Framework for Navigation
• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
• Learning to Stop: Reinforcement Learning for Efficient Patient-Level Echocardiographic Classification
• REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning
• MAPPO for Edge Server Monitoring
• Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment
• Towards Learning Boulder Excavation with Hydraulic Excavators
• Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
• Task-Oriented Communications for 3D Scene Representation: Balancing Timeliness and Fidelity
• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
• Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception
• Scalable Multi-Objective Robot Reinforcement Learning through Gradient Conflict Resolution
• Quantum Reinforcement Learning-Guided Diffusion Model for Image Synthesis via Hybrid Quantum-Classical Generative Model Architectures
• SIRAG: Towards Stable and Interpretable RAG with A Process-Supervised Multi-Agent Framework
• Near-Real-Time Resource Slicing for QoS Optimization in 5G O-RAN using Deep Reinforcement Learning
• VEGA: Electric Vehicle Navigation Agent via Physics-Informed Neural Operator and Proximal Policy Optimization
• Robust Online Residual Refinement via Koopman-Guided Dynamics Modeling
• Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
• BuildingGym: An open-source toolbox for AI-based building energy management using reinforcement learning
• Reinforcement Learning-Based Market Making as a Stochastic Control on Non-Stationary Limit Order Book Dynamics
• KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF
• Imagined Autocurricula
• Vejde: A Framework for Inductive Deep Reinforcement Learning Based on Factor Graph Color Refinement
• Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
• Fault Tolerant Control of a Quadcopter using Reinforcement Learning
• VL Norm: Rethink Loss Aggregation in RLVR
• RL Fine-Tuning Heals OOD Forgetting in SFT
• Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
• Towards a Unified View of Large Language Model Post-Training
• PPORLD-EDNetLDCT: A Proximal Policy Optimization-Based Reinforcement Learning Framework for Adaptive Low-Dose CT Denoising
• Semi-on-Demand Transit Feeders with Shared Autonomous Vehicles and Reinforcement-Learning-Based Zonal Dispatching Control
• Open Data Synthesis For Deep Research
• SABR: A Stable Adaptive Bitrate Framework Using Behavior Cloning Pretraining and Reinforcement Learning Fine-Tuning
• Machine Intelligence on the Edge: Interpretable Cardiac Pattern Localisation Using Reinforcement Learning
• PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning
• Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning
• Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities
• Energy-Efficient Learning-Based Beamforming for ISAC-Enabled V2X Networks
• HAEPO: History-Aggregated Exploratory Policy Optimization
• CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning
• SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression
• OPTIC-ER: A Reinforcement Learning Framework for Real-Time Emergency Response and Equitable Resource Allocation in Underserved African Communities
• OVSegDT: Segmenting Transformer for Open-Vocabulary Object Goal Navigation
• UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
• Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes
• Distilling Reinforcement Learning into Single-Batch Datasets
• Compass-Thinker-7B Technical Report
• PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning
• Unsupervised Skill Discovery as Exploration for Learning Agile Locomotion
• Reinforcement Learning for Large Model: A Survey
• ThinkTuning: Instilling Cognitive Reflections without Distillation
• Robust Remote Reinforcement Learning over Unreliable Communication Channels using Homomorphic State Encoding
• Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
• D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
• In-Context Reinforcement Learning via Communicative World Models
• Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
• IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
• QFOR: A Fidelity-aware Orchestrator for Quantum Computing Environments using Deep Reinforcement Learning
• HCRide: Harmonizing Passenger Fairness and Driver Preference for Human-Centered Ride-Hailing
• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience
• Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
• DiWA: Diffusion Policy Adaptation with World Models
• VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision
• D2PPO: Diffusion Policy Policy Optimization with Dispersive Loss
• Everyone Contributes! Incentivizing Strategic Cooperation in Multi-LLM Systems via Sequential Public Goods Games
• One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning
• Coordinated Humanoid Robot Locomotion with Symmetry Equivariant Reinforcement Learning Policy
• Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
• OID-PPO: Optimal Interior Design using Proximal Policy Optimization by Transforming Design Guidelines into Reward Functions
• MAO-ARAG: Multi-Agent Orchestration for Adaptive Retrieval-Augmented Generation
• Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning
• Model Predictive Adversarial Imitation Learning for Planning from Observation

Related

• Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotion
• Exploring Deep and Recurrent Architectures for Optimal Control
• Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Models
• Constrained Policy Optimization
• Learning Policies for Continuous Control via Transition Models