Meta Learning Shared Hierarchies

2017/10/26 by Kevin Frans, Jonathan Ho, Frans, Kevin +7 · 15 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotic Locomotion and Control #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.1710.09767

openalex publication_date 2017/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We develop a metalearning approach for learning hierarchically structured policies, improving sample efficiency on unseen tasks through the use of shared primitives---policies that are executed for large numbers of timesteps. Specifically, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies. We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks. We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies. We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes. We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Categorical Reparameterization with Gumbel-Softmax

Cited by

• Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning
• HiLight: A Hierarchical Reinforcement Learning Framework with Global Adversarial Guidance for Large-Scale Traffic Signal Control
• Meta-learning how to Share Credit among Macro-Actions
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Mobi-π: Mobilizing Your Robot Learning Policy
• Hyperbolic Embeddings for Learning Options in Hierarchical Reinforcement Learning
• VFunc: a Deep Generative Model for Functions
• Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
• Optimal Options for Multi-Task Reinforcement Learning Under Time Constraints
• InnateCoder: Learning Programmatic Options with Foundation Models
• Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection
• Demystifying Parallel and Distributed Deep Learning
• Discovery of Options via Meta-Learned Subgoals

Related

• Modulated Policy Hierarchies
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Hierarchical Transformers are Efficient Meta-Reinforcement Learners
• Skill-based Meta-Reinforcement Learning
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning