On the Robustness of Safe Reinforcement Learning under Observational Perturbations

2022/05/29 by Zuxin Liu, Zijian Guo, Liu, Zuxin +11 · 2 citations
Agricultural and Biological Sciences · Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cardiac electrophysiology and arrhythmias #FOS: Computer and information sciences #Insect and Pesticide Research #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2205.14691

openalex publication_date 2022/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Safe reinforcement learning (RL) trains a policy to maximize the task reward while satisfying safety constraints. While prior works focus on the performance optimality, we find that the optimal solutions of many safe RL problems are not robust and safe against carefully designed observational perturbations. We formally analyze the unique properties of designing effective observational adversarial attackers in the safe RL setting. We show that baseline adversarial attack techniques for standard RL tasks are not always effective for safe RL and propose two new approaches - one maximizes the cost and the other maximizes the reward. One interesting and counter-intuitive finding is that the maximum reward attack is strong, as it can both induce unsafe behaviors and make the attack stealthy by maintaining the reward. We further propose a robust training framework for safe RL and evaluate it via comprehensive experiments. This paper provides a pioneer work to investigate the safety and robustness of RL under observational attacks for future safe RL studies. Code is available at: \urlhttps://github.com/liuzuxin/safe-rl-robustness

Cited by

• PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning
• Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation
• Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

Related

• Certifying Safety in Reinforcement Learning under Adversarial Perturbation Attacks
• Virtuously Safe Reinforcement Learning
• Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
• Attacking and Defending Deep Reinforcement Learning Policies
• Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation