LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics

2025/04/30 by Marc Glocker, Peter Hönig, Glocker, Marc +5 · 4 citations
Computer Science · Engineering · Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Robot Manipulation and Learning #Robotic Path Planning Algorithms #Robotics (cs.RO) #Social Robot Interaction and HRI

paper · pdf · doi:10.48550/arxiv.2504.21716

openalex publication_date 2025/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present an embodied robotic system with an LLM-driven agent-orchestration architecture for autonomous household object management. The system integrates memory-augmented task planning, enabling robots to execute high-level user commands while tracking past actions. It employs three specialized agents: a routing agent, a task planning agent, and a knowledge base agent, each powered by task-specific LLMs. By leveraging in-context learning, our system avoids the need for explicit model training. RAG enables the system to retrieve context from past interactions, enhancing long-term object tracking. A combination of Grounded SAM and LLaMa3.2-Vision provides robust object detection, facilitating semantic scene understanding for task planning. Evaluation across three household scenarios demonstrates high task planning accuracy and an improvement in memory recall due to RAG. Specifically, Qwen2.5 yields best performance for specialized agents, while LLaMA3.1 excels in routing tasks. The source code is available at: https://github.com/marc1198/chat-hsr.

Cited by

• MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments
• Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
• RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systems
• Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking

Related

• KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems
• MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generation
• LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robots
• Embodied Task Planning with Large Language Models
• Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning