Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

2025/06/16 by Martin Klissarov, Akhil Bagaria, Klissarov, Martin +9 · 1 voice · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Music and Audio Processing #cs.AI

paper · pdf · doi:10.48550/arxiv.2506.14045

openalex publication_date 2025/06/16 · arxiv published 2025/06/16 · arxiv updated 2025/06/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Developing agents capable of exploring, planning and learning in complex open-ended environments is a grand challenge in artificial intelligence (AI). Hierarchical reinforcement learning (HRL) offers a promising solution to this challenge by discovering and exploiting the temporal structure within a stream of experience. The strong appeal of the HRL framework has led to a rich and diverse body of literature attempting to discover a useful structure. However, it is still not clear how one might define what constitutes good structure in the first place, or the kind of problems in which identifying it may be helpful. This work aims to identify the benefits of HRL from the perspective of the fundamental challenges in decision-making, as well as highlight its impact on the performance trade-offs of AI agents. Through these benefits, we then cover the families of methods that discover temporal structure in HRL, ranging from learning directly from online experience to offline datasets, to leveraging large language models (LLMs). Finally, we highlight the challenges of temporal structure discovery and the domains that are particularly well-suited for such endeavours.

Cited by

• Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
• ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
• Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
• Scalable Option Learning in High-Throughput Environments

Discussions

• We are looking to continue to improve this manuscript, please share your feedback! www.arxiv.org/abs/2506.14045 [bsky, 3 points, 0 comments]

Related

• Hierarchical Reinforcement Learning for Temporal Pattern Prediction
• HTMRL: Biologically Plausible Reinforcement Learning with Hierarchical Temporal Memory
• Exploring the limits of Hierarchical World Models in Reinforcement Learning
• Hierarchical Reinforcement Learning with Timed Subgoals
• Temporal-adaptive Hierarchical Reinforcement Learning