PLAS: Latent Action Space for Offline Reinforcement Learning

2020/11/14 by Wenxuan Zhou, Zhou, Wenxuan, Sujay Bajracharya +3 · 12 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO) #cs.AI #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2011.07213

arxiv created 2020/11/14 · openalex publication_date 2020/11/14 · arxiv updated 2020/11/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The goal of offline reinforcement learning is to learn a policy from a fixed dataset, without further interactions with the environment. This setting will be an increasingly more important paradigm for real-world applications of reinforcement learning such as robotics, in which data collection is slow and potentially dangerous. Existing off-policy algorithms have limited performance on static datasets due to extrapolation errors from out-of-distribution actions. This leads to the challenge of constraining the policy to select actions within the support of the dataset during training. We propose to simply learn the Policy in the Latent Action Space (PLAS) such that this requirement is naturally satisfied. We evaluate our method on continuous control benchmarks in simulation and a deformable object manipulation task with a physical robot. We demonstrate that our method provides competitive performance consistently across various continuous control tasks and different types of datasets, outperforming existing offline reinforcement learning methods with explicit constraints. Videos and code are available at https://sites.google.com/view/latent-policy.

Citations

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Prediction and Control with Temporal Segment Models
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning

Cited by

• Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
• Latent Action World Models for Control with Unlabeled Trajectories
• Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
• HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Control
• Re:Frame -- Retrieving Experience From Associative Memory
• GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
• Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
• Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL
• ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orders

Related

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
• Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• LASER: Learning a Latent Action Space for Efficient Reinforcement Learning