Learning to be Safe: Deep RL with a Safety Critic

2020/10/27 by Krishnan Srinivasan, Srinivasan, Krishnan, Benjamin Eysenbach +7 · 10 citations
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2010.14603

openalex publication_date 2020/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

Citations

• Constrained Markov Decision Processes
• Algorithms for CVaR Optimization in MDPs
• Robust Adversarial Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Distributionally Robust Reinforcement Learning

Cited by

• Achieving Safe Control Online through Integration of Harmonic Control Lyapunov-Barrier Functions with Unsafe Object-Centric Action Policies
• ProSh: Probabilistic Shielding for Model-free Reinforcement Learning
• Safety Assessment in Reinforcement Learning via Model Predictive Control
• Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
• SafeFlowMatcher: Safe and Fast Planning using Flow Matching with Control Barrier Functions
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Learning Fast, Tool aware Collision Avoidance for Collaborative Robots
• Conservative Safety Critics for Exploration
• MARS: A Meta-Adaptive Reinforcement Learning Framework for Risk-Aware Multi-Agent Portfolio Management
• Safe and Performant Controller Synthesis using Gradient-based Model Predictive Control and Control Barrier Functions

Related

• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition
• A Review of Safe Reinforcement Learning: Methods, Theory and Applications
• Reducing Safety Interventions in Provably Safe Reinforcement Learning
• Don't do it: Safer Reinforcement Learning With Rule-based Guidance
• CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions