Corruption-Robust Offline Reinforcement Learning

2021/06/11 by Zhang, Xuezhou, Chen, Yiding, Zhu, Jerry +1 · 6 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2106.06630

Abstract

We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples (s, a, r, s'), an adversary is allowed to arbitrarily modify ε fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case Ω(dε) optimality gap is unavoidable in linear MDP of dimension d, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of ε to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of ε is necessary, as we show that being adaptive to unknown ε is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.

Cited by

• Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
• Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Rates
• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning

Related

• Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption
• Corruption Robust Offline Reinforcement Learning with Human Feedback
• Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
• Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
• Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning