Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

2019/10/01 by Xue Bin Peng, Aviral Kumar, Peng, Xue Bin +5 · 76 citations
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Advanced Control Systems Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1910.00177

openalex publication_date 2019/10/01 · openalex created_date 2019/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

Cited by

• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization
• Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
• Action-Sufficient Goal Representations
• Generative Actor Critic
• VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
• What Happens Next? Next Scene Prediction with a Unified Video Model
• Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
• Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment
• TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
• Scalable Offline Model-Based RL with Action Chunks
• Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
• PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modeling
• FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments
• Real-World Reinforcement Learning of Active Perception Behaviors
• When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• Video Generation Models Are Good Latent Reward Models
• Efficient Diffusion Planning with Temporal Diffusion
• Escaping the Verifier: Learning to Reason via Demonstrations
• Learning Massively Multitask World Models for Continuous Control
• SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
• π^*_0.6: a VLA That Learns From Experience
• Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
• Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL
• Multi-agent Coordination via Flow Matching
• Leveraging Discrete Function Decomposability for Scientific Design
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
• Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
• RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning
• LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values
• Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback
• ADPO: Anchored Direct Preference Optimization
• Fine-tuning Flow Matching Generative Models with Intermediate Feedback
• RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning
• Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learning
• MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control
• VIST3A: Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator
• Emotional Cognitive Modeling Framework with Desire-Driven Objective Optimization for LLM-empowered Agent in Social Simulation
• Human-in-the-Loop Bandwidth Estimation for Quality of Experience Optimization in Real-Time Video Communication
• Offline Reinforcement Learning with Generative Trajectory Policies
• Real-Time Motion-Controllable Autoregressive Video Diffusion
• DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
• Value Flows
• Primal-Dual Direct Preference Optimization for Constrained LLM Alignment
• Adjusting the Output of Decision Transformer with Action Gradient
• A KL-regularization Framework for Learning to Plan with Adaptive Priors
• Flexible Locomotion Learning with Diffusion Model Predictive Control
• HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Control
• RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
• Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
• TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
• Adaptive Margin RLHF via Preference over Preferences
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Adaptive Policy Backbone via Shared Network
• Actor-Critic without Actor
• RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
• Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
• Direct Preference Optimization for Speech Autoregressive Diffusion Models
• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
• Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
• Let's Roleplay: Examining LLM Alignment in Collaborative Dialogues
• Grasp-MPC: Closed-Loop Visual Grasping via Value-Guided Model Predictive Control
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPs
• The Geometry of Nonlinear Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning
• A Reward-Directed Diffusion Framework for Generative Design Optimization
• Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
• Weighted Conditional Flow Matching
• Agentic Reinforced Policy Optimization
• Test-time Offline Reinforcement Learning on Goal-related Experience

Related

• Q-Value Weighted Regression: Reinforcement Learning with Limited Data
• CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
• Adaptive Trade-Offs in Off-Policy Learning
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• Skill or Luck? Return Decomposition via Advantage Functions