SPoRt -- Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL

2025/04/08 by Cloete, Jacques, Vertovec, Nikolaus, Abate, Alessandro · 3 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2504.06386

Abstract

To apply reinforcement learning to safety-critical applications, we ought to provide safety guarantees during both policy training and deployment. In this work, we present theoretical results that place a bound on the probability of violating a safety property for a new task-specific policy in a model-free, episodic setting. This bound, based on a maximum policy ratio computed with respect to a 'safe' base policy, can also be applied to temporally-extended properties (beyond safety) and to robust control problems. To utilize these results, we introduce SPoRt, which provides a data-driven method for computing this bound for the base policy using the scenario approach, and includes Projected PPO, a new projection-based approach for training the task-specific policy while maintaining a user-specified bound on property violation. SPoRt thus enables users to trade off safety guarantees against task-specific performance. Complementing our theoretical results, we present experimental results demonstrating this trade-off and comparing the theoretical bound to posterior bounds derived from empirical violation rates.

Citations

• Embedding Safety into RL: A New Take on Trust Region Methods
• Finite sample learning of moving targets
• Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
• The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems
• Online Shielding for Reinforcement Learning
• Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships
• A Review of Reinforcement Learning for Autonomous Building Energy Management
• Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control
• Certified Reinforcement Learning with Logic Guidance
• Learning agile and dynamic motor skills for legged robots
• Metrics for Signal Temporal Logic Formulae
• A Lyapunov-based Approach to Safe Reinforcement Learning
• Safe Reinforcement Learning on Autonomous Vehicles
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Rewriting System for Convex Optimization Problems
• A rewriting system for convex optimization problems
• Safe Reinforcement Learning via Shielding
• Proximal Policy Optimization Algorithms
• Constrained Policy Optimization
• CVXPY: A Python-Embedded Modeling Language for Convex Optimization
• Prioritized Experience Replay
• Quantitative Approximation of the Probability Distribution of a Markov Process by Formal Abstractions
• Trust Region Policy Optimization
• A Sampling-and-Discarding Approach to Chance-Constrained Optimization: Feasibility and Optimality
• Robustness of temporal logic specifications for continuous-time signals
• Reinforcement Learning: An Introduction

Related

• Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
• Provably Optimal Reinforcement Learning under Safety Filtering
• Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
• Towards Safe Reinforcement Learning with a Safety Editor Policy
• Safety Aware Reinforcement Learning (SARL)