Hierarchical Reinforcement Learning with Hindsight

2018/05/21 by Andrew Levy, Levy, Andrew, Robert W. Platt +3 · 3 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.1805.08180

openalex publication_date 2018/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functions and hindsight learning, allowing agents to learn policies belonging to different time scales in parallel. We show that our method significantly accelerates learning in a variety of discrete and continuous tasks.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Continuous control with deep reinforcement learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Strategic Attentive Writer for Learning Macro-Actions
• Hindsight Experience Replay

Cited by

• SCOPE: Language Models as One-Time Teacher for Hierarchical Planning in Text Environments
• Generalizable Hierarchical Skill Learning via Object-Centric Representation
• A Unified Framework for Zero-Shot Reinforcement Learning

Related

• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Generalized Hindsight for Reinforcement Learning
• Reducing Commitment to Tasks with Off-Policy Hierarchical Reinforcement Learning
• Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation