Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
2023/02/03 by Zihao Wang, Shaofei Cai, Wang, Zihao +8 · 79 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2302.01560
openalex publication_date 2023/02/03 · openalex created_date 2023/02/08 · openalex updated_date 2026/07/28
Abstract
We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose "\underlineDescribe, \underlineExplain, \underlinePlan and \underlineSelect" (DEPS), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated plan by integrating description of the plan execution process and providing self-explanation of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal selector, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the ObtainDiamond grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.
Cited by
- Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex
- GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
- SCOPE: Language Models as One-Time Teacher for Hierarchical Planning in Text Environments
- Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning
- LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems
- HalluClean: A Unified Framework to Combat Hallucinations in LLMs
- Improving Cooperation in Collaborative Embodied AI
- Temporal Blindness in Multi-Turn LLM Agents: Misaligned Tool Use vs. Human Time Perception
- Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
- Experience-Driven Exploration for Efficient API-Free AI Agents
- EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making
- MIMIC: Integrating Diverse Personality Traits for Better Game Testing Using Large Language Model
- Drift No More? Context Equilibria in Multi-Turn LLM Interactions
- ReInAgent: A Context-Aware GUI Agent Enabling Human-in-the-Loop Mobile Task Navigation
- Information Seeking for Robust Decision Making under Partial Observability
- When Should Users Check? A Decision-Theoretic Model of Confirmation Frequency in Multi-Step AI Agent Tasks
- SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planning
- PhysiAgent: An Embodied Agent Framework in Physical World
- LAGEA: Language Guided Embodied Agents for Robotic Manipulation
- Creative Adversarial Testing (CAT): A Novel Framework for Evaluating Goal-Oriented Agentic AI Systems
- AutoMem: Automated Learning of Memory as a Cognitive Skill
- Code Driven Planning with Domain-Adaptive Critic
- MCTS-EP: Empowering Embodied Planning with Online Preference Optimization
- LEED: A Highly Efficient and Scalable LLM-Empowered Expert Demonstrations Framework for Multi-Agent Reinforcement Learning
- From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft
- Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution
- TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making
- World Model Implanting for Test-time Adaptation of Embodied Agents
- Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models
- VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft
- Virtual Community: An Open World for Humans, Robots, and Society
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
- LOOP: A Plug-and-Play Neuro-Symbolic Framework for Enhancing Planning in Autonomous Systems
- Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
- Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
- \(X\)-evolve: Solution space evolution powered by large language models
- UrzaGPT: LoRA-Tuned Large Language Models for Card Selection in Collectible Card Games
- Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimization
- The Term 'Agent' Has Been Diluted Beyond Utility and Requires Redefinition
- ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"
- FlowForge: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold
- Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems
- Conditional Multi-Stage Failure Recovery for Embodied Agents
- The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems
- A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
- Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulation
- Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
- Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
- GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System
- Leveraging In-Context Learning for Language Model Agents
- Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safety
- Conversational Interfaces for Parametric Conceptual Architectural Design: Integrating Mixed Reality with LLM-driven Interaction
- TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models
- DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments
- CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic
- MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
- EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM
- Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
- Probe by Gaming: A Game-based Benchmark for Assessing Conceptual Knowledge in LLMs
- The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
- BAR: A Backward Reasoning based Agent for Complex Minecraft Tasks
- Cost-Awareness in Tree-Search LLM Planning: A Systematic Study
- G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
- InnateCoder: Learning Programmatic Options with Foundation Models
- Reasoning Capabilities of Large Language Models on Dynamic Tasks
- Deliberate Planning in Language Models with Symbolic Representation
- Requesting Expert Reasoning: Augmenting LLM Agents with Learned Collaborative Intervention
- UAV-VLN: End-to-End Vision Language guided Navigation for UAVs
- Gated Coordination for Efficient Multi-Agent Collaboration in Minecraft Game
- PRISM: Perception Reasoning Interleaved for Sequential Decision Making
- Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability
- What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agents
- When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail
- Credible Plan-Driven RAG Method for Multi-Hop Question Answering
- WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents
- Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction
- Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models
- SEAL: Steerable Reasoning Calibration of Large Language Models for Free
Related