Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences

2015/06/12 by Hongyuan Mei, Mohit Bansal, Mei, Hongyuan +3 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Robotics (cs.RO) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1506.04089

openalex publication_date 2015/06/12 · openalex created_date 2019/07/30 · openalex updated_date 2026/07/28

Abstract

We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.

Citations

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Recurrent Neural Network Regularization
• Long Short-Term Memory
• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Recurrent Models of Visual Attention

Cited by

• SIMA 2: A Generalist Embodied Agent for Virtual Worlds
• SIGVerse: A cloud-based VR platform for research on social and embodied human-robot interaction
• Mind Your Outliers! Investigating the Negative Impact of Outliers on\n Active Learning for Visual Question Answering
• A Narration-based Reward Shaping Approach using Grounded Natural\n Language Commands
• Grounding Complex Navigational Instructions Using Scene Graphs
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning
• Mapping Natural Language Instructions to Mobile UI Action Sequences
• Executing Instructions in Situated Collaborative Interactions

Related

• Plan, Attend, Generate: Character-level Neural Machine Translation with\n Planning in the Decoder
• Deep Reinforcement Learning For Sequence to Sequence Models
• Sequence-to-Sequence Learning with Latent Neural Grammars
• Sequence Generation: From Both Sides to the Middle
• Lexicon Learning for Few-Shot Neural Sequence Modeling