VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

2019/10/18 by Luisa Zintgraf, Kyriacos Shiarlis, Zintgraf, Luisa +11 · 1 voice · 10 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #Gaussian Processes and Bayesian Inference #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.08348

arxiv published 2019/10/18 · arxiv updated 2020/02/27

Abstract

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.

Citations

• Contextual Markov Decision Processes
• Auto-Encoding Variational Bayes
• Planning and acting in partially observable stochastic domains
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic\n Context Variables
• Meta-learning of Sequential Strategies
• Meta reinforcement learning as task inference

Cited by

• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learning
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Heuristic Transformer: Belief Augmented In-Context Reinforcement Learning
• Social World Model-Augmented Mechanism Design Policy Learning
• Information-Theoretic Policy Pre-Training with Empowerment
• STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
• Adaptive Policy Backbone via Shared Network
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning\n without Sacrifices
• Exploitation Is All You Need... for Exploration

Discussions

• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning [hn, 3 points, 0 comments]

Related

• ContraBAR: Contrastive Bayes-Adaptive Deep RL
• Offline Meta Learning of Exploration
• Deep Interactive Bayesian Reinforcement Learning via Meta-Learning
• Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
• Meta-Learning with Variational Bayes