Robust Model Predictive Shielding for Safe Reinforcement Learning with Stochastic Dynamics

2019/10/24 by Shuo Li, Osbert Bastani, Li, Shuo +1 · 6 citations
Computer Science · Physics and Astronomy · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1910.10885

openalex publication_date 2019/10/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper proposes a framework for safe reinforcement learning that can handle stochastic nonlinear dynamical systems. We focus on the setting where the nominal dynamics are known, and are subject to additive stochastic disturbances with known distribution. Our goal is to ensure the safety of a control policy trained using reinforcement learning, e.g., in a simulated environment. We build on the idea of model predictive shielding (MPS), where a backup controller is used to override the learned policy as needed to ensure safety. The key challenge is how to compute a backup policy in the context of stochastic dynamics. We propose to use a tube-based robust NMPC controller as the backup controller. We estimate the tubes using sampled trajectories, leveraging ideas from statistical learning theory to obtain high-probability guarantees. We empirically demonstrate that our approach can ensure safety in stochastic systems, including cart-pole and a non-holonomic particle with random obstacles.

Citations

• Safe Model-based Reinforcement Learning with Stability Guarantees
• Learning dexterous in-hand manipulation

Cited by

• Safety Assessment in Reinforcement Learning via Model Predictive Control
• CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
• What You Don't Know Can Hurt You: How Well do Latent Safety Filters Understand Partially Observable Safety Constraints?
• Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
• Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control
• Parallel-Constraint Model Predictive Control: Exploiting Parallel Computation for Improving Safety

Related

• Safe Reinforcement Learning with Nonlinear Dynamics via Model Predictive Shielding
• Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part I - Stochastic case
• Probabilistically safe and efficient model-based reinforcement learning
• Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning