Boosting Offline Reinforcement Learning with Residual Generative Modeling

2021/06/19 by Hua Wei, Wei, Hua, Deheng Ye +13 · 1 citation
Computer Science · Economics, Econometrics and Finance · #68T01 #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #FOS: Computer and information sciences #I.2.1 #I.2.8 #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Sports Analytics and Performance

paper · pdf · doi:10.48550/arxiv.2106.10411

openalex publication_date 2021/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline reinforcement learning (RL) tries to learn the near-optimal policy with recorded offline experience without online exploration. Current offline RL research includes: 1) generative modeling, i.e., approximating a policy using fixed data; and 2) learning the state-action value function. While most research focuses on the state-action function part through reducing the bootstrapping error in value function approximation induced by the distribution shift of training data, the effects of error propagation in generative modeling have been neglected. In this paper, we analyze the error in generative modeling. We propose AQL (action-conditioned Q-learning), a residual generative model to reduce policy approximation error for offline RL. We show that our method can learn more accurate policy approximations in different benchmark datasets. In addition, we show that the proposed offline RL method can learn more competitive AI agents in complex control tasks under the multiplayer online battle arena (MOBA) game Honor of Kings.

Citations

• Playing Atari with Deep Reinforcement Learning
• Continuous control with deep reinforcement learning
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Challenges of Real-World Reinforcement Learning
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Behavior Regularized Offline Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning

Cited by

• DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty

Related

• An Optimistic Perspective on Offline Reinforcement Learning
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
• Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
• Model-based Offline Reinforcement Learning with Local Misspecification
• A Minimalist Approach to Offline Reinforcement Learning