Offline RL Policies Should be Trained to be Adaptive

2022/07/05 by Dibya Ghosh, Anurag Ajay, Ghosh, Dibya +5 · 10 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2207.02200

ICML 2022 (long talk)

arxiv created 2022/07/05 · openalex publication_date 2022/07/05 · arxiv updated 2022/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline RL algorithms must account for the fact that the dataset they are provided may leave many facets of the environment unknown. The most common way to approach this challenge is to employ pessimistic or conservative methods, which avoid behaviors that are too dissimilar from those in the training dataset. However, relying exclusively on conservatism has drawbacks: performance is sensitive to the exact degree of conservatism, and conservative objectives can recover highly suboptimal policies. In this work, we propose that offline RL methods should instead be adaptive in the presence of uncertainty. We show that acting optimally in offline RL in a Bayesian sense involves solving an implicit POMDP. As a result, optimal policies for offline RL must be adaptive, depending not just on the current state but rather all the transitions seen so far during evaluation.We present a model-free algorithm for approximating this optimal adaptive policy, and demonstrate the efficacy of learning such adaptive policies in offline RL benchmarks.

Cited by

• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measures
• Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learning
• Automated Construction of Artificial Lattice Structures with Designer Electronic States
• Behavioral Exploration: Learning to Explore via In-Context Adaptation
• Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning
• Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning
• Decision Flow Policy Optimization
• Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning

Related

• Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
• Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
• COMBO: Conservative Offline Model-Based Policy Optimization
• DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
• Automatic Trade-off Adaptation in Offline RL