2023/04/26 by Renhao Wang, Wang, Renhao, Jiayuan Mao +9 · 6 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain (mathematical analysis) #Executable #FOS: Computer and information sciences #Generalization #Human–computer interaction #Leverage (statistics) #Machine learning #Modular design #Multimodal Machine Learning Applications #Natural language processing #Parsing #Pipeline (software) #Programming language #Reinforcement Learning in Robotics #Robot #Robot Manipulation and Learning #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2304.13826
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/04/26 · openalex created_date 2023/04/30 · openalex updated_date 2026/07/28
Robots operating in the real world require both rich manipulation skills as well as the ability to semantically reason about when to apply those skills. Towards this goal, recent works have integrated semantic representations from large-scale pretrained vision-language (VL) models into manipulation models, imparting them with more general reasoning capabilities. However, we show that the conventional pretraining-finetuning pipeline for integrating such representations entangles the learning of domain-specific action information and domain-general visual information, leading to less data-efficient training and poor generalization to unseen objects and tasks. To this end, we propose ProgramPort, a modular approach to better leverage pretrained VL models by exploiting the syntactic and semantic structures of language instructions. Our framework uses a semantic parser to recover an executable program, composed of functional modules grounded on vision and action across different modalities. Each functional module is realized as a combination of deterministic computation and learnable neural networks. Program execution produces parameters to general manipulation primitives for a robotic end-effector. The entire modular network can be trained with end-to-end imitation learning objectives. Experiments show that our model successfully disentangles action and perception, translating to improved zero-shot and compositional generalization in a variety of manipulation behaviors. Project webpage at: \urlhttps://progport.github.io.