Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

2022/09/14 by Ma, Xiaoteng, Liang, Zhipeng, Blanchet, Jose +5 · 5 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2209.06620

Abstract

Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy is deployed) and the training environment (e.g., a simulator). This paper attempts to address these issues simultaneously with distributionally robust offline RL, where we learn a distributionally robust policy using historical data obtained from the source environment by optimizing against a worst-case perturbation thereof. In particular, we move beyond tabular settings and consider linear function approximation. More specifically, we consider two settings, one where the dataset is well-explored and the other where the dataset has sufficient coverage of the optimal policy. We propose two algorithms~-- one for each of the two settings~-- that achieve error bounds O(d^1/2/N^1/2) and O(d^3/2/N^1/2) respectively, where d is the dimension in the linear function approximation and N is the number of trajectories in the dataset. To the best of our knowledge, they provide the first non-asymptotic results of the sample complexity in this setting. Diverse experiments are conducted to demonstrate our theoretical findings, showing the superiority of our algorithm against the non-robust one.

Cited by

• Online Robust Reinforcement Learning with General Function Approximation
• Provably Efficient Sample Complexity for Robust CMDP
• Distributionally Robust Self Paced Curriculum Reinforcement Learning
• Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning

Related

• Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
• Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Online Robust Reinforcement Learning with General Function Approximation
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation