Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning

2025/06/26 by Hwang, Jaebak, Sanghyeon Lee, Lee, Sanghyeon +4
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2506.21039

openalex publication_date 2025/06/26 · openalex created_date 2025/10/15 · openalex updated_date 2026/08/01

Abstract

Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial solutions, their reliance on conventional hindsight relabeling often fails to correct subgoal infeasibility, leading to inefficient high-level planning. To address this, we propose Strict Subgoal Execution (SSE), a graph-based hierarchical RL framework that integrates Frontier Experience Replay (FER) to separate unreachable from admissible subgoals and streamline high-level decision making. FER delineates the reachability frontier using failure and partial-success transitions, which identifies unreliable subgoals, increases subgoal reliability, and reduces unnecessary high-level decisions. Additionally, SSE employs a decoupled exploration policy to cover underexplored regions of the goal space and a path refinement that adjusts edge costs using observed low-level failures. Experimental results across diverse long-horizon benchmarks show that SSE consistently outperforms existing goal-conditioned and hierarchical RL methods in both efficiency and success rate. Our code is available at https://jaebak1996.github.io/SSE/

Citations

• Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning
• CQM: Curriculum Reinforcement Learning with a Quantized World Model
• FoX: Formation-aware exploration in multi-agent reinforcement learning
• Imitating Graph-Based Planning with Goal-Conditioned Policies
• DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
• Goal-Conditioned Reinforcement Learning: Problems and Solutions
• Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement\n Learning
• Discovering and Achieving Goals via World Models
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals
• A Max-Min Entropy Framework for Reinforcement Learning
• Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey
• Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for\n Sample-Efficient Exploration
• Planning with Goal-Conditioned Policies
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Mapping State Space using Landmarks for Universal Goal Reaching
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• Exploration via Hindsight Goal Generation
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Near-Optimal Representation Learning for Hierarchical Reinforcement\n Learning
• Unsupervised Control Through Non-Parametric Discriminative Rewards
• Exploration by Random Network Distillation
• Visual Reinforcement Learning with Imagined Goals
• Data-Efficient Hierarchical Reinforcement Learning
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and\n Request for Research
• Addressing Function Approximation Error in Actor-Critic Methods
• Diversity is All You Need: Learning Skills without a Reward Function
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Hindsight Experience Replay
• FeUdal Networks for Hierarchical Reinforcement Learning
• The Option-Critic Architecture
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Prioritized Experience Replay
• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• A Note on Two Problems in Connexion with Graphs

Related

• Active Hierarchical Exploration with Stable Subgoal Representation Learning
• Learning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
• Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs
• Landmark-Guided Subgoal Generation in Hierarchical Reinforcement\n Learning
• Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks