Safe and Near-Optimal Control with Online Dynamics Learning

2025/09/20 by Manish Prajapat, Prajapat, Manish, Johannes Köhler +5
Computer Science · Decision Sciences · #AI-based Problem Solving and Planning #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Robotic Path Planning Algorithms #Robotics (cs.RO) #Simulation Techniques and Applications #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.16650

openalex publication_date 2025/09/20 · openalex created_date 2025/10/16 · openalex updated_date 2026/08/01

Abstract

Achieving both optimality and safety under unknown system dynamics is a central challenge in real-world deployment of agents. To address this, we introduce a notion of maximum safe dynamics learning, where sufficient exploration is performed within the space of safe policies. Our method executes pessimistically safe policies while optimistically exploring informative states and, despite not reaching them due to model uncertainty, ensures continuous online learning of dynamics. The framework achieves first-of-its-kind results: learning the dynamics model sufficiently - up to an arbitrary small tolerance (subject to noise) - in a finite time, while ensuring provably safe operation throughout with high probability and without requiring resets. Building on this, we propose an algorithm to maximize rewards while learning the dynamics only to the extent needed to achieve close-to-optimal performance. Unlike typical reinforcement learning (RL) methods, our approach operates online in a non-episodic setting and ensures safety throughout the learning process. We demonstrate the effectiveness of our approach in challenging domains such as autonomous car racing and drone navigation under aerodynamic effects - scenarios where safety is critical and accurate modeling is difficult.

Citations

• Finite-Sample-Based Reachability for Safe Control with Gaussian Process Dynamics
• Safe exploration in reproducing kernel Hilbert spaces
• Stochastic Model Predictive Control for Sub-Gaussian Noise
• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
• Towards safe and tractable Gaussian process-based MPC: Efficient sampling within a sequential quadratic programming framework
• On Safety in Safe Bayesian Optimization
• Truly No-Regret Learning in Constrained MDPs
• Safe Guaranteed Exploration for Non-linear Systems
• Lipschitz and Hölder Continuity in Reproducing Kernel Hilbert Spaces
• Robust feedback motion planning via contraction theory
• Submodular Reinforcement Learning
• Near-Optimal Multi-Agent Learning for Safe Coverage Control
• Safe Exploration and Escape Local Minima with Model Predictive Control under Partially Unknown Constraints
• Constrained Policy Optimization via Bayesian World Models
• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
• Safe Continuous Control with Constrained Model-Based Policy Optimization
• On Information Gain and Regret Bounds in Gaussian Process Bandits
• Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework
• Safe Reinforcement Learning in Constrained Markov Decision Processes
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• Exploration-Exploitation in Constrained MDPs
• acados: a modular open-source framework for fast embedded optimal control
• AMZ Driverless: The Full Autonomous Racing System
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Stagewise Safe Bayesian Optimization with Gaussian Processes
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Learning-based Model Predictive Control for Safe Exploration
• Safe Reinforcement Learning via Shielding
• Lower Bounds on Regret for Noisy Gaussian Process Bandit Optimization
• Constrained Policy Optimization
• Safe Model-based Reinforcement Learning with Stability Guarantees
• On Kernelized Multi-armed Bandits
• Bayesian Optimization with Safety Constraints: Safe and Automatic\n Parameter Tuning in Robotics
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Constrained Markov Decision Processes

Related

• Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework
• Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions
• Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)
• Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach
• Computationally and Sample Efficient Safe Reinforcement Learning Using Adaptive Conformal Prediction