Provable Hierarchy-Based Meta-Reinforcement Learning

2021/10/18 by Kurtland Chua, Qi Lei, Chua, Kurtland +3
Computer Science · Engineering · #Adaptive Dynamic Programming Control #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2110.09507

openalex publication_date 2021/10/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors. However, existing work either assume access to expert-constructed hierarchies, or use hierarchy-learning heuristics with no provable guarantees. To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task. We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics. Analogous to supervised meta-learning theory, we provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.

Citations

• Trust Region Policy Optimization
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Exploration--Exploitation in MDPs with Options
• Data-Efficient Hierarchical Reinforcement Learning
• Learning Multi-Level Hierarchies with Hindsight
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Few-Shot Learning via Learning the Representation, Provably
• Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisited

Related

• Hierarchical Transformers are Efficient Meta-Reinforcement Learners
• Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptation
• Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning