2022/10/14 by Peihao Chen, Dongyu Ji, Chen, Peihao +11 · 22 citations
Computer Science · #Artificial intelligence #Benchmark (surveying) #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Granularity #Human–computer interaction #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Object (grammar) #Path (computing) #Programming language #Real-time computing #Representation (politics) #Robot #Semantic mapping #Task (project management) #Waypoint #cs.CV
paper · pdf · doi:10.48550/arxiv.2210.07506
published in arXiv (Cornell University) (Cornell University) · Accepted by NeurIPS 2022
arxiv created 2022/10/14 · openalex publication_date 2022/10/14 · arxiv updated 2022/10/17 · openalex created_date 2022/10/19 · openalex updated_date 2026/07/28
We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (e.g., color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. Code is available at https://github.com/PeihaoChen/WS-MGMap.