Sub-policy Adaptation for Hierarchical Reinforcement Learning

2019/06/13 by Alexander C. Li, Carlos Florensa, Li, Alexander C. +5 · 2 citations
Computer Science · #Adaptive Dynamic Programming Control #Advanced Multi-Objective Optimization Algorithms #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1906.05862

openalex publication_date 2019/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method for training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and results are available at sites.google.com/view/hippo-rl

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• FeUdal Networks for Hierarchical Reinforcement Learning
• Data-Efficient Hierarchical Reinforcement Learning
• Learning dexterous in-hand manipulation

Cited by

• Policy Compatible Skill Incremental Learning via Lazy Learning Interface
• Scalable Option Learning in High-Throughput Environments

Related

• Hierarchical Reinforcement Learning with Advantage-Based Auxiliary\n Rewards
• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Temporal-adaptive Hierarchical Reinforcement Learning
• Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
• Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization