Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration

2019/11/26 by Stepputtis, Simon, Campbell, Joseph, Phielipp, Mariano +2 · 1 citation
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · doi:10.48550/arxiv.1911.11744

Abstract

In this work we propose a novel end-to-end imitation learning approach which combines natural language, vision, and motion information to produce an abstract representation of a task, which in turn is used to synthesize specific motion controllers at run-time. This multimodal approach enables generalization to a wide variety of environmental conditions and allows an end-user to direct a robot policy through verbal communication. We empirically validate our approach with an extensive set of simulations and show that it achieves a high task success rate over a variety of conditions while remaining amenable to probabilistic interpretability.

Cited by

• Learning to Generate Pointing Gestures in Situated Embodied Conversational Agents

Related

• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Language Conditioned Imitation Learning over Unstructured Data
• A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
• Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
• Robotic Imitation of Human Actions