Dynamic Memory Networks for Visual and Textual Question Answering

2016/03/04 by Caiming Xiong, Stephen Merity, Xiong, Caiming +3 · 1 voice · 37 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Multimodal Machine Learning Applications #Topic Modeling #cs.CL #cs.CV #cs.NE

paper · pdf · doi:10.48550/arxiv.1603.01417

openalex publication_date 2016/03/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.

Citations

• VQA: Visual Question Answering

Cited by

• Visual Question Answering: A Survey of Methods and Datasets
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Location-aware Graph Convolutional Networks for Video Question Answering
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Learning to activate logic rules for textual reasoning
• Iterative Visual Reasoning Beyond Convolutions
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Uncertainty based Class Activation Maps for Visual Question Answering
• Pedestrian Attribute Recognition: A Survey
• URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs
• Dual Attention Networks for Multimodal Reasoning and Matching
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
• Think Visually: Question Answering through Virtual Imagery
• Learning Operations on a Stack with Neural Turing Machines
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Attention on Abstract Visual Reasoning
• Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition
• Transfer Learning in Visual and Relational Reasoning
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Pay Attention to Those Sets! Learning Quantification from Images
• Compositional Attention Networks for Machine Reasoning
• Incorporating Glosses into Neural Word Sense Disambiguation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Hierarchical Memory Networks for Answer Selection on Unknown Words
• Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields
• EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction
• Attention-Aware Face Hallucination via Deep Reinforcement Learning
• Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Pre-Trained Models: Past, Present and Future
• Visual Question Answering with Memory-Augmented Networks

Discussions

• Dynamic Memory Networks for Visual and Textual Question Answering [hn, 122 points, 15 comments]

Related

• Visual Question Answering with Memory-Augmented Networks
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Textually Enriched Neural Module Networks for Visual Question Answering
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Ask Me Even More: Dynamic Memory Tensor Networks (Extended Model)