Multitask Adaptation by Retrospective Exploration with Learned World\n Models

2021/10/25 by Artem Zholus, Aleksandr I. Panov, Zholus, Artem +1
Computer Science · #Anomaly Detection Techniques and Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2110.13241

openalex publication_date 2021/10/25 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Model-based reinforcement learning (MBRL) allows solving complex tasks in a\nsample-efficient manner. However, no information is reused between the tasks.\nIn this work, we propose a meta-learned addressing model called RAMa that\nprovides training samples for the MBRL agent taken from continuously growing\ntask-agnostic storage. The model is trained to maximize the expected agent's\nperformance by selecting promising trajectories solving prior tasks from the\nstorage. We show that such retrospective exploration can accelerate the\nlearning process of the MBRL agent by better informing learned dynamics and\nprompting agent with exploratory trajectories. We test the performance of our\napproach on several domains from the DeepMind control suite, from Metaworld\nmultitask benchmark, and from our bespoke environment implemented with a\nrobotic NVIDIA Isaac simulator to test the ability of the model to act in a\nphotorealistic, ray-traced environment.\n

Citations

• Trust Region Policy Optimization
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Model-Free Episodic Control
• Neural Episodic Control
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Recurrent World Models Facilitate Policy Evolution
• Data Valuation using Reinforcement Learning
• The Ingredients of Real-World Robotic Reinforcement Learning
• Object Goal Navigation using Goal-Oriented Semantic Exploration

Related

• Human-Timescale Adaptation in an Open-Ended Task Space
• A Model-based Approach for Sample-efficient Multi-task Reinforcement\n Learning
• Meta-Learning to Explore via Memory Density Feedback
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Leveraging Temporally Extended Behavior Sharing for Multi-task Reinforcement Learning