Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

2024/08/29 by Toshinori Kitamura, Kitamura, Toshinori, Tadashi Kozuno +15 · 4 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2408.16286

openalex publication_date 2024/08/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Designing a safe policy for uncertain environments is crucial in real-world control systems. However, this challenge remains inadequately addressed within the Markov decision process (MDP) framework. This paper presents the first algorithm guaranteed to identify a near-optimal policy in a robust constrained MDP (RCMDP), where an optimal policy minimizes cumulative cost while satisfying constraints in the worst-case scenario across a set of environments. We first prove that the conventional policy gradient approach to the Lagrangian max-min formulation can become trapped in suboptimal solutions. This occurs when its inner minimization encounters a sum of conflicting gradients from the objective and constraint functions. To address this, we leverage the epigraph form of the RCMDP problem, which resolves the conflict by selecting a single gradient from either the objective or the constraints. Building on the epigraph form, we propose a bisection search algorithm with a policy gradient subroutine and prove that it identifies an ε-optimal policy in an RCMDP with O(ε^-4) robust policy evaluations.

Cited by

• Provably Efficient Sample Complexity for Robust CMDP
• Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPs
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality

Related

• Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees
• A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes
• Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets
• Policy Gradient for Robust Markov Decision Processes
• Best Policy Identification in Linear MDPs