Curriculum in Gradient-Based Meta-Reinforcement Learning

2020/02/19 by Bhairav Mehta, Tristan Deleu, Mehta, Bhairav +8 · 8 citations
Computer Science · Mathematics · #Adaptation (eye) #Artificial Intelligence (cs.AI) #Artificial intelligence #Artificial neural network #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generalization #Leverage (statistics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine learning #Mathematics #Meta learning (computer science) #Overfitting #Reinforcement Learning in Robotics #Reinforcement learning #Stability (learning theory) #Task (project management) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.07956

published in arXiv (Cornell University) (Cornell University) · 11 pages, 10 figures

arxiv created 2020/02/19 · openalex publication_date 2020/02/19 · arxiv updated 2020/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Gradient-based meta-learners such as Model-Agnostic Meta-Learning (MAML) have shown strong few-shot performance in supervised and reinforcement learning settings. However, specifically in the case of meta-reinforcement learning (meta-RL), we can show that gradient-based meta-learners are sensitive to task distributions. With the wrong curriculum, agents suffer the effects of meta-overfitting, shallow adaptation, and adaptation instability. In this work, we begin by highlighting intriguing failure cases of gradient-based meta-RL and show that task distributions can wildly affect algorithmic outputs, stability, and performance. To address this problem, we leverage insights from recent literature on domain randomization and propose meta Active Domain Randomization (meta-ADR), which learns a curriculum of tasks for gradient-based meta-RL in a similar as ADR does for sim2real transfer. We show that this approach induces more stable policies on a variety of simulated locomotion and navigation tasks. We assess in- and out-of-distribution generalization and find that the learned task distributions, even in an unstructured task space, greatly improve the adaptation performance of MAML. Finally, we motivate the need for better benchmarking in meta-RL that prioritizes generalization over single-task adaption performance.

Citations

• Learning to reinforcement learn
• Stein Variational Policy Gradient
• Reverse Curriculum Generation for Reinforcement Learning
• The effects of negative adaptation in Model-Agnostic Meta-Learning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

Cited by

• Improving Generalization in Meta-RL with Imaginary Tasks from Latent\n Dynamics Mixture

Related

• Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling
• On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning
• Model-based Adversarial Meta-Reinforcement Learning
• Train Hard, Fight Easy: Robust Meta Reinforcement Learning
• From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning