2023/12/17 by Xiangtong Yao, Yao, Xiangtong, Hongkuan Zhou +18 · 4 citations
Computer Science · #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Robotics (cs.RO) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2312.10807
openalex publication_date 2023/12/17 · openalex created_date 2023/12/20 · openalex updated_date 2026/07/28
Language-conditioned robot manipulation is an emerging field aimed at enabling seamless communication and cooperation between humans and robotic agents by teaching robots to comprehend and execute instructions conveyed in natural language. This interdisciplinary area integrates scene understanding, language processing, and policy learning to bridge the gap between human instructions and robot actions. In this comprehensive survey, we systematically explore recent advancements in language-conditioned robot manipulation. We categorize existing methods based on the primary ways language is integrated into the robot system, namely language for state evaluation, language as a policy condition, language for cognitive planning and reasoning, and language in unified vision-language-action models. Specifically, we further analyze state-of-the-art techniques from five axes of action granularity, data and supervision regimes, system cost and latency, environments and evaluations, and task specification. Additionally, we highlight the key debates in the field. Finally, we discuss open challenges and future research directions, focusing on potentially enhancing generalization capabilities and addressing safety issues in language-conditioned robot manipulators.