Online Robust Reinforcement Learning with Model Uncertainty

2021/09/29 by Yue Wang, Wang, Yue, Shaofeng Zou +1 · 17 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.14523

Accepted by NeurIPS 2021

openalex publication_date 2021/09/29 · arxiv created 2021/10/28 · arxiv updated 2021/10/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Robust reinforcement learning (RL) is to find a policy that optimizes the worst-case performance over an uncertainty set of MDPs. In this paper, we focus on model-free robust RL, where the uncertainty set is defined to be centering at a misspecified MDP that generates a single sample trajectory sequentially and is assumed to be unknown. We develop a sample-based approach to estimate the unknown uncertainty set and design a robust Q-learning algorithm (tabular case) and robust TDC algorithm (function approximation setting), which can be implemented in an online and incremental fashion. For the robust Q-learning algorithm, we prove that it converges to the optimal robust Q function, and for the robust TDC algorithm, we prove that it converges asymptotically to some stationary points. Unlike the results in [Roy et al., 2017], our algorithms do not need any additional conditions on the discount factor to guarantee the convergence. We further characterize the finite-time error bounds of the two algorithms and show that both the robust Q-learning and robust TDC algorithms converge as fast as their vanilla counterparts(within a constant factor). Our numerical experiments further demonstrate the robustness of our algorithms. Our approach can be readily extended to robustify many other algorithms, e.g., TD, SARSA, and other GTD algorithms.

Citations

• Robust Markov Decision Processes
• Robust Dynamic Programming
• Finite-Sample Analysis of Proximal Gradient TD Algorithms
• Adversarial Attacks on Neural Network Policies
• Robust Adversarial Reinforcement Learning
• Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning
• Wasserstein Robust Reinforcement Learning
• Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samples
• Finite-Sample Analysis for SARSA with Linear Function Approximation
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Robust Reinforcement Learning with Wasserstein Constraint
• Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
• Robust Reinforcement Learning using Adversarial Populations
• Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexity
• Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithms
• Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation

Cited by

• Online Robust Reinforcement Learning with General Function Approximation
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• Provably Efficient Sample Complexity for Robust CMDP
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets
• ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty
• Online Bayesian Risk-Averse Reinforcement Learning
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning
• DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
• Qualitative Analysis of ω-Regular Objectives on Robust MDPs
• Hybrid Cross-domain Robust Reinforcement Learning
• Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees
• Distributionally Robust Deep Q-Learning
• Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning
• Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysis
• Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss

Related

• Robust Reinforcement Learning using Offline Data
• On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm
• Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation
• Online Robust Reinforcement Learning with General Function Approximation
• Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity