Constrained Reinforcement Learning Under Model Mismatch

2024/05/02 by Zhongchang Sun, Sun, Zhongchang, Sihong He +5 · 2 citations
Computer Science · #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2405.01327

openalex publication_date 2024/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing studies on constrained reinforcement learning (RL) may obtain a well-performing policy in the training environment. However, when deployed in a real environment, it may easily violate constraints that were originally satisfied during training because there might be model mismatch between the training and real environments. To address the above challenge, we formulate the problem as constrained RL under model uncertainty, where the goal is to learn a good policy that optimizes the reward and at the same time satisfy the constraint under model mismatch. We develop a Robust Constrained Policy Optimization (RCPO) algorithm, which is the first algorithm that applies to large/continuous state space and has theoretical guarantees on worst-case reward improvement and constraint violation at each iteration during the training. We demonstrate the effectiveness of our algorithm on a set of RL tasks with constraints.

Cited by

• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality

Related

• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Robust Constrained Reinforcement Learning
• Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
• Resilient Constrained Reinforcement Learning