LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

2024/06/19 by Huang, Zhe, Pohovey, John, Yammanuru, Ananya +1 · 3 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2406.13787

Abstract

Large Language Models (LLM) and Vision Language Models (VLM) enable robots to ground natural language prompts into control actions to achieve tasks in an open world. However, when applied to a long-horizon collaborative task, this formulation results in excessive prompting for initiating or clarifying robot actions at every step of the task. We propose Language-driven Intention Tracking (LIT), leveraging LLMs and VLMs to model the human user's long-term behavior and to predict the next human intention to guide the robot for proactive collaboration. We demonstrate smooth coordination between a LIT-based collaborative robot and the human user in collaborative cooking tasks.

Cited by

• Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance
• Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models
• Hierarchical Intention Tracking with Switching Trees for Real-Time Adaptation to Dynamic Human Intentions during Collaboration

Related

• Using Natural Language for Human-Robot Collaboration in the Real World
• Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration
• Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
• LLM A*: Human in the Loop Large Language Models Enabled A* Search for Robotics
• From Vague Instructions to Task Plans: A Feedback-Driven HRC Task Planning Framework based on LLMs