RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents

2024/02/06 by Tomoyuki Kagaya, Thong Jing Yuan, Kagaya, Tomoyuki +15 · 22 citations
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2402.03610

openalex publication_date 2024/02/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.

Cited by

• In-Context Distillation with Self-Consistency Cascades: A Simple, Training-Free Way to Reduce LLM Agent Costs
• Test-Time Adaptation for LLM Agents via Environment Interaction
• Cache Mechanism for Agent RAG Systems
• Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
• ReCode: Unify Plan and Action for Universal Granularity Control
• NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoning
• How²: How to learn from procedural How-to questions
• Towards Proprioception-Aware Embodied Planning for Dual-Arm Humanoid Robots
• ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
• Beyond Manuals and Tasks: Instance-Level Context Learning for LLM Agents
• ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learning
• Memory Transfer Planning: LLM-driven Context-Aware Code Adaptation for Robot Manipulation
• DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
• ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaboration
• D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents
• Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
• EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer
• H²R: Hierarchical Hindsight Reflection for Multi-Task LLM Agents
• Human Centric General Physical Intelligence for Agile Manufacturing Automation
• A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real World
• GoalfyMax: A Protocol-Driven Multi-Agent System for Intelligent Experience Entities
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs

Related

• TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision
• P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task
• InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning
• Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG
• Intrinsic Memory Agents: Heterogeneous Multi-Agent LLM Systems through Structured Contextual Memory