H²R: Hierarchical Hindsight Reflection for Multi-Task LLM Agents

2025/09/16 by S. Ye, Ye, Shicheng, Chao Yu +6 · 1 citation
Computer Science · Decision Sciences · #Anomaly Detection Techniques and Applications #Construct (python library) #Decoupling (probability) #Generalization #Hindsight bias #Knowledge base #Knowledge transfer #Reflection (computer programming) #Scheme (mathematics) #Scientific Computing and Data Management #Time Series Analysis and Forecasting

paper · pdf · doi:10.48550/arxiv.2509.12810

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/09/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/08/05

Abstract

Large language model (LLM)-based agents have shown strong potential in multi-task scenarios, owing to their ability to transfer knowledge across diverse tasks. However, existing approaches often treat prior experiences and knowledge as monolithic units, leading to inefficient and coarse-grained knowledge transfer. In this work, we propose a novel hierarchical memory architecture that enables fine-grained knowledge transfer by decoupling high-level planning memory from low-level execution memory. To construct and refine these hierarchical memories, we introduce Hierarchical Hindsight Reflection (H²R), a mechanism that distills reusable and hierarchical knowledge from past agent-environment interactions. At test time, H²R performs retrievals of high-level and low-level memories separately, allowing LLM-based agents to efficiently access and utilize task-relevant knowledge for new tasks.Experimental results across two benchmarks demonstrate that H²R can improve generalization and decision-making performance, outperforming prior baselines such as Expel.

Citations

• How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior
• A Survey of LLM-based Agents in Medicine: How far are we from Baymax?
• MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systems
• Enhancing Agent Learning through World Dynamics Modeling
• A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges
• AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning
• A Survey on Large Language Model-Based Game Agents
• AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents
• RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents
• Understanding the planning of LLM agents: A survey
• Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• The Faiss library
• Retrieval-Augmented Generation for Large Language Models: A Survey
• ExpeL: LLM Agents Are Experiential Learners
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Re2G: Retrieve, Rerank, Generate
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Reinforcement learning

Cited by

• MemEvolve: Meta-Evolution of Agent Memory Systems

Related

• HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Model
• Coarse-to-Fine Grounded Memory for LLM Agent Planning
• Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents
• G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems
• Efficiently Enhancing General Agents With Hierarchical-categorical Memory