Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

2022/06/12 by Qinbo Bai, Amrit Singh Bedi, Bai, Qinbo +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning and Algorithms #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2206.05850

openalex publication_date 2022/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider the problem of constrained Markov decision process (CMDP) in continuous state-actions spaces where the goal is to maximize the expected cumulative reward subject to some constraints. We propose a novel Conservative Natural Policy Gradient Primal-Dual Algorithm (C-NPG-PD) to achieve zero constraint violation while achieving state of the art convergence results for the objective value function. For general policy parametrization, we prove convergence of value function to global optimal upto an approximation error due to restricted policy class. We even improve the sample complexity of existing constrained NPG-PD algorithm \citeDing2020 from O(1/ε⁶) to O(1/ε⁴). To the best of our knowledge, this is the first work to establish zero constraint violation with Natural policy gradient style algorithms for infinite horizon discounted CMDPs. We demonstrate the merits of proposed algorithm via experimental evaluations.

Cited by

• Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs
• Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithm
• Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shifts

Related

• Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
• Double Duality: Variational Primal-Dual Policy Optimization for Constrained Reinforcement Learning
• A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
• Learning Policies with Zero or Bounded Constraint Violation for\n Constrained MDPs
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning