Improving Generalization in Meta Reinforcement Learning using Learned Objectives

2019/10/09 by Louis Kirsch, Sjoerd van Steenkiste, Kirsch, Louis +3 · 19 citations
Computer Science · Mathematics · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #cs.AI #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.04098

Accepted to ICLR 2020

openalex publication_date 2019/10/09 · arxiv created 2020/02/14 · arxiv updated 2020/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Citations

• Auto-Encoding Variational Bayes
• Learning to Optimize
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Policy Distillation
• Learning to Optimize Neural Nets
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Recurrent World Models Facilitate Policy Evolution
• Meta-learning curiosity algorithms

Cited by

• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Uniform Priors for Data-Efficient Transfer
• Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces
• Model-based Adversarial Meta-Reinforcement Learning
• How Should We Meta-Learn Reinforcement Learning Algorithms?
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Evolving Reinforcement Learning Algorithms
• What Can Learned Intrinsic Rewards Capture?
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Grounding Artificial Intelligence in the Origins of Human Behavior
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent\n Dynamics Mixture
• Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Policy Gradient RL Algorithms as Directed Acyclic Graphs
• LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
• Discovering Reinforcement Learning Algorithms

Related

• Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
• How Should We Meta-Learn Reinforcement Learning Algorithms?
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• RL³: Boosting Meta Reinforcement Learning via RL inside RL²
• On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning