A Provable Approach for End-to-End Safe Reinforcement Learning

2025/05/28 by Akifumi Wachi, Wachi, Akifumi, Kohei Miyaguchi +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Information Theory (cs.IT) #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2505.21852

openalex publication_date 2025/05/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A longstanding goal in safe reinforcement learning (RL) is a method to ensure the safety of a policy throughout the entire process, from learning to operation. However, existing safe RL paradigms inherently struggle to achieve this objective. We propose a method, called Provably Lifetime Safe RL (PLS), that integrates offline safe RL with safe policy deployment to address this challenge. Our proposed method learns a policy offline using return-conditioned supervised learning and then deploys the resulting policy while cautiously optimizing a limited set of parameters, known as target returns, using Gaussian processes (GPs). Theoretically, we justify the use of GPs by analyzing the mathematical relationship between target and actual returns. We then prove that PLS finds near-optimal target returns while guaranteeing safety with high probability. Empirically, we demonstrate that PLS outperforms baselines both in safety and reward performance, thereby achieving the longstanding goal to obtain high rewards while ensuring the safety of a policy throughout the lifetime from learning to operation.

Citations

• A Survey of Constraint Formulations in Safe Reinforcement Learning
• Sample Path Regularity of Gaussian Processes from the Covariance Kernel
• Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
• Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms
• Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
• Datasets and Benchmarks for Offline Safe Reinforcement Learning
• Safe Offline Reinforcement Learning with Real-Time Budget Constraints
• Training Diffusion Models with Reinforcement Learning
• Constrained Decision Transformer for Offline Safe Reinforcement Learning
• When does return-conditioned supervised learning work for offline reinforcement learning?
• A Review of Safe Reinforcement Learning: Methods, Theory and Applications
• COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
• Training language models to follow instructions with human feedback
• Safe Reinforcement Learning by Imagining the Near Future
• Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees
• RvS: What is Essential for Offline RL via Supervised Learning?
• Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Benchmarks for Deep Off-Policy Evaluation
• Safe Reinforcement Learning in Constrained Markov Decision Processes
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Verifiably Safe Exploration for End-to-End Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Reward-Conditioned Policies
• Training Agents using Upside-Down Reinforcement Learning
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• Safe Policies for Reinforcement Learning via Primal-Dual Methods
• Reinforcement Learning in Healthcare: A Survey
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• Batch Policy Learning under Constraints
• Off-Policy Deep Reinforcement Learning without Exploration
• Stagewise Safe Bayesian Optimization with Gaussian Processes
• Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning
• Safe Reinforcement Learning via Shielding
• Constrained Policy Optimization
• Safe Model-based Reinforcement Learning with Stability Guarantees
• On Kernelized Multi-armed Bandits
• Concrete Problems in AI Safety
• End-to-End Training of Deep Visuomotor Policies
• Effects of Safety State Augmentation on Safe Exploration

Cited by

• Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control

Related

• Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve\n Optimality?
• Probabilistic Shielding for Safe Reinforcement Learning
• Provably Optimal Reinforcement Learning under Safety Filtering
• Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
• Safe Distributional Reinforcement Learning