Improving Generalization in Meta Reinforcement Learning using Learned Objectives

2019/10/09 by Louis Kirsch, Sjoerd van Steenkiste, Kirsch, Louis +3 · 6 citations
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1910.04098

openalex publication_date 2019/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Citations

• Auto-Encoding Variational Bayes
• Learning to Optimize
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Policy Distillation
• Learning to Optimize Neural Nets
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Recurrent World Models Facilitate Policy Evolution
• Meta-learning curiosity algorithms

Cited by

• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Uniform Priors for Data-Efficient Transfer
• Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces
• Model-based Adversarial Meta-Reinforcement Learning

Related

• Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees
• How Should We Meta-Learn Reinforcement Learning Algorithms?
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• RL³: Boosting Meta Reinforcement Learning via RL inside RL²
• On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning