Safe Guaranteed Dynamics Exploration with Probabilistic Models

2025/09/20 by Prajapat, Manish, Köhler, Johannes, Zeilinger, Melanie N. +1
#Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2509.16650

Abstract

Ensuring both optimality and safety is critical for the real-world deployment of agents, but becomes particularly challenging when the system dynamics are unknown. To address this problem, we introduce a notion of maximum safe dynamics learning via sufficient exploration in the space of safe policies. We propose a pessimistically safe framework that optimistically explores informative states and, despite not reaching them due to model uncertainty, ensures continuous online learning of dynamics. The framework achieves first-of-its-kind results: learning the dynamics model sufficiently - up to an arbitrary small tolerance (subject to noise) - in a finite time, while ensuring provably safe operation throughout with high probability and without requiring resets. Building on this, we propose an algorithm to maximize rewards while learning the dynamics only to the extent needed to achieve close-to-optimal performance. Unlike typical reinforcement learning (RL) methods, our approach operates online in a non-episodic setting and ensures safety throughout the learning process. We demonstrate the effectiveness of our approach in challenging domains such as autonomous car racing and drone navigation under aerodynamic effects - scenarios where safety is critical and accurate modeling is difficult.

Citations

• Finite-Sample-Based Reachability for Safe Control with Gaussian Process Dynamics
• Safe exploration in reproducing kernel Hilbert spaces
• Stochastic Model Predictive Control for Sub-Gaussian Noise
• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
• Towards safe and tractable Gaussian process-based MPC: Efficient sampling within a sequential quadratic programming framework
• On Safety in Safe Bayesian Optimization
• Truly No-Regret Learning in Constrained MDPs
• Safe Guaranteed Exploration for Non-linear Systems
• Lipschitz and Hölder Continuity in Reproducing Kernel Hilbert Spaces
• Robust feedback motion planning via contraction theory
• Submodular Reinforcement Learning
• Near-Optimal Multi-Agent Learning for Safe Coverage Control
• Safe Exploration and Escape Local Minima with Model Predictive Control under Partially Unknown Constraints
• Constrained Policy Optimization via Bayesian World Models
• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
• Safe Continuous Control with Constrained Model-Based Policy Optimization
• On Information Gain and Regret Bounds in Gaussian Process Bandits
• Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework
• Safe Reinforcement Learning in Constrained Markov Decision Processes
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Exploration-Exploitation in Constrained MDPs
• acados: a modular open-source framework for fast embedded optimal\n control
• AMZ Driverless: The Full Autonomous Racing System
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Stagewise Safe Bayesian Optimization with Gaussian Processes
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Learning-based Model Predictive Control for Safe Exploration
• Safe Reinforcement Learning via Shielding
• Lower Bounds on Regret for Noisy Gaussian Process Bandit Optimization
• Constrained Policy Optimization
• Safe Model-based Reinforcement Learning with Stability Guarantees
• On Kernelized Multi-armed Bandits
• Bayesian Optimization with Safety Constraints: Safe and Automatic\n Parameter Tuning in Robotics
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Constrained Markov Decision Processes

Related

• Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework
• Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions
• Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)
• Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach
• Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Prediction