Offline RL With Resource Constrained Online Deployment

2021/10/07 by Jayanth Reddy Regatti, Regatti, Jayanth Reddy, Aniket Anand Deshmukh +8
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2110.03165

openalex publication_date 2021/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Offline reinforcement learning is used to train policies in scenarios where real-time access to the environment is expensive or impossible. As a natural consequence of these harsh conditions, an agent may lack the resources to fully observe the online environment before taking an action. We dub this situation the resource-constrained setting. This leads to situations where the offline dataset (available for training) can contain fully processed features (using powerful language models, image models, complex sensors, etc.) which are not available when actions are actually taken online. This disconnect leads to an interesting and unexplored problem in offline RL: Is it possible to use a richly processed offline dataset to train a policy which has access to fewer features in the online environment? In this work, we introduce and formalize this novel resource-constrained problem setting. We highlight the performance gap between policies trained using the full offline dataset and policies trained using limited features. We address this performance gap with a policy transfer algorithm which first trains a teacher agent using the offline dataset where features are fully available, and then transfers this knowledge to a student agent that only uses the resource-constrained features. To better capture the challenge of this setting, we propose a data collection procedure: Resource Constrained-Datasets for RL (RC-D4RL). We evaluate our transfer algorithm on RC-D4RL and the popular D4RL benchmarks and observe consistent improvement over the baseline (TD3+BC without transfer). The code for the experiments is available at https://github.com/JayanthRR/RC-OfflineRL.

Citations

• Playing Atari with Deep Reinforcement Learning
• Planning and acting in partially observable stochastic domains
• Continuous control with deep reinforcement learning
• Domain Generalization by Marginal Transfer Learning
• A Generalization Error Bound for Multi-class Domain Generalization
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Behavior Regularized Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Knowledge Distillation: A Survey
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• A Minimalist Approach to Offline Reinforcement Learning

Related

• MOORe: Model-based Offline-to-Online Reinforcement Learning
• Augmenting Offline RL with Unlabeled Data
• Data Valuation for Offline Reinforcement Learning
• Efficient Online Reinforcement Learning with Offline Data
• Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions