2020/04/30 by Arjun Majumdar, Ayush Shrivastava, Majumdar, Arjun +9 · 10 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2004.14973
openalex publication_date 2020/04/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Following a navigation instruction such as 'Walk down the stairs and stop at\nthe brown sofa' requires embodied AI agents to ground scene elements referenced\nvia language (e.g. 'stairs') to visual content in the environment (pixels\ncorresponding to 'stairs').\n We ask the following question -- can we leverage abundant 'disembodied'\nweb-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn\nvisual groundings (what do 'stairs' look like?) that improve performance on a\nrelatively data-starved embodied perception task (Vision-and-Language\nNavigation)? Specifically, we develop VLN-BERT, a visiolinguistic\ntransformer-based model for scoring the compatibility between an instruction\n('...stop at the brown sofa') and a sequence of panoramic RGB images captured\nby the agent. We demonstrate that pretraining VLN-BERT on image-text pairs from\nthe web before fine-tuning on embodied path-instruction data significantly\nimproves performance on VLN -- outperforming the prior state-of-the-art in the\nfully-observed setting by 4 absolute percentage points on success rate.\nAblations of our pretraining curriculum show each stage to be impactful -- with\ntheir combination resulting in further positive synergistic effects.\n