Hierarchical Reinforcement Learning with Hindsight

2018/05/21 by Andrew Levy, Robert W. Platt, Robert Platt +4 · 11 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Reinforcement Learning in Robotics #Robotics (cs.RO) #cs.AI #cs.LG #cs.NE #cs.RO #stat.ML

paper · pdf · doi:10.48550/arxiv.1805.08180

Duplicate. See arXiv:1712.00948 "Learning Multi-Level Hierarchies with Hindsight" for latest version

openalex publication_date 2018/05/21 · arxiv created 2019/03/08 · arxiv updated 2019/03/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functions and hindsight learning, allowing agents to learn policies belonging to different time scales in parallel. We show that our method significantly accelerates learning in a variety of discrete and continuous tasks.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Continuous control with deep reinforcement learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Strategic Attentive Writer for Learning Macro-Actions
• Hindsight Experience Replay
• Learning Multi-Level Hierarchies with Hindsight

Cited by

• SCOPE: Language Models as One-Time Teacher for Hierarchical Planning in Text Environments
• Generalizable Hierarchical Skill Learning via Object-Centric Representation
• A Unified Framework for Zero-Shot Reinforcement Learning
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• Mapping State Space using Landmarks for Universal Goal Reaching
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Efficient Robotic Policy Learning via Latent Space Backward Planning
• HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning
• A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
• Learning Compositional Neural Programs for Continuous Control
• Correcting Experience Replay for Multi-Agent Communication

Related

• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Generalized Hindsight for Reinforcement Learning
• Reducing Commitment to Tasks with Off-Policy Hierarchical Reinforcement Learning
• Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation