Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation

2024/05/02 by Shangding Gu, Bilgehan Sel, Gu, Shangding +11 · 5 citations
Engineering · Health Professions · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Occupational Health and Safety Research #Safety Systems Engineering in Autonomy

paper · pdf · doi:10.48550/arxiv.2405.01677

openalex publication_date 2024/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Ensuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we aim to address this conflicting relation by leveraging the theory of gradient manipulation. Initially, we analyze the conflict between reward and safety gradients. Subsequently, we tackle the balance between reward and safety optimization by proposing a soft switching policy optimization method, for which we provide convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. Finally, we evaluate the effectiveness of our method on the Safety-MuJoCo Benchmark and a popular safe RL benchmark, Omnisafe. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization.

Cited by

• KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
• Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signals
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
• Proactive Constrained Policy Optimization with Preemptive Penalty

Related

• Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
• Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
• Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning
• Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation
• Towards Safe Reinforcement Learning with a Safety Editor Policy