Language Conditioned Imitation Learning over Unstructured Data

2020/05/15 by Corey Lynch, Pierre Sermanet, Lynch, Corey +1 · 29 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Robot Manipulation and Learning #Robotics (cs.RO) #cs.AI #cs.CL #cs.CV #cs.RO

paper · pdf · doi:10.48550/arxiv.2005.07648

Published at RSS 2021

openalex publication_date 2020/05/15 · arxiv created 2021/07/07 · arxiv updated 2021/07/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot. Prior work in imitation learning typically requires each task be specified with a task id or goal image -- something that is often impractical in open-world environments. On the other hand, previous approaches in instruction following allow agent behavior to be guided by language, but typically assume structure in the observations, actuators, or language that limit their applicability to complex settings like robotics. In this work, we present a method for incorporating free-form natural language conditioning into imitation learning. Our approach learns perception from pixels, natural language understanding, and multitask continuous control end-to-end as a single neural network. Unlike prior work in imitation learning, our method is able to incorporate unlabeled and unstructured demonstration data (i.e. no task or language labels). We show this dramatically improves language conditioned performance, while reducing the cost of language annotation to less than 1% of total data. At test time, a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment, specified only with natural language descriptions of each task (e.g. "open the drawer...now pick up the block...now press the green button..."). To scale up the number of instructions an agent can follow, we propose combining text conditioned policies with large pretrained neural language models. We find this allows a policy to be robust to many out-of-distribution synonym instructions, without requiring new demonstrations. See videos of a human typing live text commands to our agent at language-play.github.io

Citations

• Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
• Grounded Language Learning in a Simulated 3D World
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• BabyAI: A Platform to Study the Sample Efficiency of Grounded Language\n Learning

Cited by

• SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding
• Language-Conditioned Safe Trajectory Generation for Spacecraft Rendezvous
• Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
• QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents
• IPR-1: Interactive Physical Reasoner
• Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
• EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations
• DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning
• From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priors
• End-to-end Listen, Look, Speak and Act
• Memory Transfer Planning: LLM-driven Context-Aware Code Adaptation for Robot Manipulation
• LAGEA: Language Guided Embodied Agents for Robotic Manipulation
• Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
• Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policy
• AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
• Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation
• Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Control
• Structured Task Solving via Modular Embodied Intelligence: A Case Study on Rubik's Cube
• Unified Vision-Language-Action Model
• RationalVLA: A Rational Vision-Language-Action Model with Dual System
• BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learning
• Learning Compositional Behaviors from Demonstration and Language
• L2D2: Robot Learning from 2D Drawings
• Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions
• Efficient Robotic Policy Learning via Latent Space Backward Planning
• CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation
• SAS-Prompt: Large Language Models as Numerical Optimizers for Robot Self-Improvement
• Language-Critique Imitation Learning from Suboptimal Demonstrations
• DFM-VLA: Iterative Action Refinement for Robot Manipulation via Discrete Flow Matching

Related

• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data
• What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data
• Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
• Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning