Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks

2025/11/10 by Hao Wang, Sathwik Karnik, Wang, Hao +4
Computer Science · #AI-based Problem Solving and Planning #Robotic Path Planning Algorithms #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.07410

Abstract

Large Language Models (LLMs) and Vision Language Models (VLMs) have become popular tools for embodied high-level planning. However, their deployment in black-box settings often leads to unpredictable or costly errors. To harness their capabilities more reliably in robotic systems, we empirically investigate practical strategies for integrating language models as closed-loop planners. Concretely, we study how the control horizon and warm-starting impact the performance of language model-based planners. We design and conduct controlled experiments to extract actionable insights, providing recommendations that can help improve the performance and robustness of language model-based embodied planning. The full implementation and experiments are available on the project website

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
• Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation
• Guiding Long-Horizon Task and Motion Planning with Vision Language Models
• ReplanVLM: Replanning Robotic Tasks with Visual Language Models
• Robotic Control via Embodied Chain-of-Thought Reasoning
• Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following
• LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning
• RePLan: Robotic Replanning with Perception and Language Models
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
• Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
• Grounding Classical Task Planners via Vision-Language Models
• GPT-4 Technical Report
• PaLM-E: An Embodied Multimodal Language Model
• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
• Code as Policies: Language Model Programs for Embodied Control
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Related

• Can LLM-Reasoning Models Replace Classical Planning? A Benchmark Study
• Language models are robotic planners: reframing plans as goal refinement graphs
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
• A Systematic Study of Large Language Models for Task and Motion Planning With PDDLStream
• Translating Natural Language to Planning Goals with Large-Language Models