Watch, Try, Learn: Meta-Learning from Demonstrations and Reward

2019/06/07 by Allan Zhou, Eric Jang, Zhou, Allan +17 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1906.03352

openalex publication_date 2019/06/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from both demonstrations and trial-and-error experience with sparse reward feedback. In comparison to meta-imitation, this approach enables the agent to effectively and efficiently improve itself autonomously beyond the demonstration data. In comparison to meta-reinforcement learning, we can scale to substantially broader distributions of tasks, as the demonstration reduces the burden of exploration. Our experiments show that our method significantly outperforms prior approaches on a set of challenging, vision-based control tasks.

Citations

• Reinforcement learning in robotics: A survey
• Learning to reinforcement learn
• RL²: Fast Reinforcement Learning via Slow Reinforcement Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Evolved Policy Gradients
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic\n Context Variables
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Deep Q-learning from Demonstrations

Cited by

• Balance Equation-based Distributionally Robust Offline Imitation Learning

Related

• Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
• Self-Imitation Learning from Demonstrations
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning
• Learning from Imperfect Demonstrations from Agents with Varying Dynamics
• Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker