COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

2024/09/23 by Kehui Liu, Zixin Tang, Liu, Kehui +9 · 10 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Modular Robots and Swarm Intelligence #Robotics (cs.RO) #Robotics and Automated Systems #Service-Oriented Architecture and Web Services

paper · pdf · doi:10.48550/arxiv.2409.15146

openalex publication_date 2024/09/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Leveraging the powerful reasoning capabilities of large language models (LLMs), recent LLM-based robot task planning methods yield promising results. However, they mainly focus on single or multiple homogeneous robots on simple tasks. Practically, complex long-horizon tasks always require collaboration among multiple heterogeneous robots especially with more complex action spaces, which makes these tasks more challenging. To this end, we propose COHERENT, a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems including quadrotors, robotic dogs, and robotic arms. Specifically, a Proposal-Execution-Feedback-Adjustment (PEFA) mechanism is designed to decompose and assign actions for individual robots, where a centralized task assigner makes a task planning proposal to decompose the complex task into subtasks, and then assigns subtasks to robot executors. Each robot executor selects a feasible action to implement the assigned subtask and reports self-reflection feedback to the task assigner for plan adjustment. The PEFA loops until the task is completed. Moreover, we create a challenging heterogeneous multi-robot task planning benchmark encompassing 100 complex long-horizon tasks. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency. The experimental videos, code, and benchmark are released at https://github.com/MrKeee/COHERENT.

Cited by

• LLM-Based Generalizable Hierarchical Task Planning and Execution for Heterogeneous Robot Teams with Event-Driven Replanning
• Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence
• RoboOS-NeXT: A Unified Memory-based Framework for Lifelong, Scalable, and Robust Multi-Robot Collaboration
• LLM-Grounded Dynamic Task Planning with Hierarchical Temporal Logic for Human-Aware Multi-Robot Handover
• RobotFleet: An Open-Source Framework for Centralized Multi-Robot Task Planning
• Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
• SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents
• Multi-Robot Task Planning for Multi-Object Retrieval Tasks with Distributed On-Site Knowledge via Large Language Models
• AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
• DEXTER-LLM: Dynamic and Explainable Coordination of Multi-Robot Systems in Unknown Environments via Large Language Models

Related

• SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models
• HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models
• LLM-Based Generalizable Hierarchical Task Planning and Execution for Heterogeneous Robot Teams with Event-Driven Replanning
• MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models
• RoCo: Dialectic Multi-Robot Collaboration with Large Language Models