Soft-Robust Actor-Critic Policy-Gradient

2018/03/11 by Esther Derman, Daniel J. Mankowitz, Derman, Esther +6 · 4 citations
Computer Science · Decision Sciences · Engineering · Mathematics · #Advanced Memory and Neural Computing #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #Simulation Techniques and Applications #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1803.04848

UAI 2018

openalex publication_date 2018/03/11 · arxiv created 2018/10/24 · arxiv updated 2018/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robust Reinforcement Learning aims to derive optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering the worst case scenario, robust policies can be overly conservative. Our soft-robust framework is an attempt to overcome this issue. In this paper, we present a novel Soft-Robust Actor-Critic algorithm (SR-AC). It learns an optimal policy with respect to a distribution over an uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust strategies. We show the convergence of SR-AC and test the efficiency of our approach on different domains by comparing it against regular learning methods and their robust formulations.

Citations

• Playing Atari with Deep Reinforcement Learning
• Robust Dynamic Programming
• Continuous control with deep reinforcement learning
• Reinforcement Learning under Model Mismatch

Cited by

• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
• DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
• An empirical investigation of the challenges of real-world reinforcement learning

Related

• DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
• Policy Gradient Method For Robust Reinforcement Learning
• Natural Actor-Critic for Robust Reinforcement Learning with Function Approximation
• Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning
• Entropic Risk Constrained Soft-Robust Policy Optimization