2020/11/07 by Peter Anderson, Ayush Shrivastava, Anderson, Peter +11 · 21 citations
Computer Science · Engineering · Mathematics · #Action (physics) #Artificial intelligence #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain (mathematical analysis) #Domain Adaptation and Few-Shot Learning #Engineering #FOS: Computer and information sciences #Human–computer interaction #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language #Robot #Robotics #Robotics (cs.RO) #Space (punctuation) #Task (project management) #Transfer (computing) #cs.CL #cs.CV #cs.RO
paper · pdf · doi:10.48550/arxiv.2011.03807
published in arXiv (Cornell University), 671-681 (Cornell University) · CoRL 2020
arxiv created 2020/11/07 · openalex publication_date 2020/11/07 · arxiv updated 2020/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study the challenging problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions. Recent work on the task of Vision-and-Language Navigation (VLN) has achieved significant progress in simulation. To assess the implications of this work for robotics, we transfer a VLN agent trained in simulation to a physical robot. To bridge the gap between the high-level discrete action space learned by the VLN agent, and the robot's low-level continuous action space, we propose a subgoal model to identify nearby waypoints, and use domain randomization to mitigate visual domain differences. For accurate sim and real comparisons in parallel environments, we annotate a 325m2 office space with 1.3km of navigation instructions, and create a digitized replica in simulation. We find that sim-to-real transfer to an environment not seen in training is successful if an occupancy map and navigation graph can be collected and annotated in advance (success rate of 46.8% vs. 55.9% in sim), but much more challenging in the hardest setting with no prior mapping at all (success rate of 22.5%).