Constrained Decision Transformer for Offline Safe Reinforcement Learning

2023/02/14 by Zuxin Liu, Zijian Guo, Liu, Zuxin +11 · 7 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2302.07351

openalex publication_date 2023/02/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Safe reinforcement learning (RL) trains a constraint satisfaction policy by interacting with the environment. We aim to tackle a more challenging problem: learning a safe policy from an offline dataset. We study the offline safe RL problem from a novel multi-objective optimization perspective and propose the ε-reducible concept to characterize problem difficulties. The inherent trade-offs between safety and task performance inspire us to propose the constrained decision transformer (CDT) approach, which can dynamically adjust the trade-offs during deployment. Extensive experiments show the advantages of the proposed method in learning an adaptive, safe, robust, and high-reward policy. CDT outperforms its variants and strong offline safe RL baselines by a large margin with the same hyperparameters across all tasks, while keeping the zero-shot adaptation capability to different constraint thresholds, making our approach more suitable for real-world RL under constraints. The code is available at https://github.com/liuzuxin/OSRL.

Cited by

• V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functions
• Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
• Online Optimization for Offline Safe Reinforcement Learning
• Boundary-to-Region Supervision for Offline Safe Reinforcement Learning
• Implicit Safety Alignment from Crowd Preferences
• Temporal Logic-Based Multi-Vehicle Backdoor Attacks against Offline RL Agents in End-to-end Autonomous Driving
• Generative Auto-Bidding in Large-Scale Competitive Auctions via Diffusion Completer-Aligner

Related

• Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
• Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning
• Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
• Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
• Online Optimization for Offline Safe Reinforcement Learning