Dynamic Memory Networks for Visual and Textual Question Answering

2016/03/04 by Caiming Xiong, Stephen Merity, Xiong, Caiming +3 · 1 voice · 595 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Computer hardware #Computer science #Dynamic random-access memory #Multimodal Machine Learning Applications #Natural language processing #Question answering #Semiconductor memory #Topic Modeling #cs.CL #cs.CV #cs.NE

paper · pdf · doi:10.48550/arxiv.1603.01417

published in arXiv (Cornell University) (Cornell University)

arxiv created 2016/03/04 · openalex publication_date 2016/03/04 · arxiv updated 2016/03/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.

Citations

• VQA: Visual Question Answering

Cited by

• Visual Question Answering: A Survey of Methods and Datasets
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models
• Location-aware Graph Convolutional Networks for Video Question Answering
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Learning to activate logic rules for textual reasoning
• Iterative Visual Reasoning Beyond Convolutions
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Uncertainty based Class Activation Maps for Visual Question Answering
• Pedestrian Attribute Recognition: A Survey
• URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs
• Dual Attention Networks for Multimodal Reasoning and Matching
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
• Think Visually: Question Answering through Virtual Imagery
• Learning Operations on a Stack with Neural Turing Machines
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Attention on Abstract Visual Reasoning
• Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition
• Transfer Learning in Visual and Relational Reasoning
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Pay Attention to Those Sets! Learning Quantification from Images
• Compositional Attention Networks for Machine Reasoning
• Incorporating Glosses into Neural Word Sense Disambiguation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Hierarchical Memory Networks for Answer Selection on Unknown Words
• Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields
• EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction
• Attention-Aware Face Hallucination via Deep Reinforcement Learning
• Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Pre-Trained Models: Past, Present and Future
• Visual Question Answering with Memory-Augmented Networks
• High-Order Attention Models for Visual Question Answering
• Learning Invariants through Soft Unification
• Unsupervised Learning from Continuous Video in a Scalable Predictive Recurrent Network
• Multimodal Machine Learning: A Survey and Taxonomy
• Integrating both Visual and Audio Cues for Enhanced Video Caption
• Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Recent Trends in Deep Learning Based Natural Language Processing
• Visual Reference Resolution using Attention Memory for Visual Dialog
• A simple neural network module for relational reasoning
• Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
• Creativity: Generating Diverse Questions using Variational Autoencoders
• The emergent algebraic structure of RNNs and embeddings in NLP
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Zero-Shot Transfer VQA Dataset
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• Multimodal Memory Modelling for Video Captioning
• DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification
• DeepLogic: Towards End-to-End Differentiable Logical Reasoning
• Hierarchical Memory Networks
• Attention on Attention: Architectures for Visual Question Answering (VQA)
• MarioQA: Answering Questions by Watching Gameplay Videos
• Multimodal Machine Learning: A Survey and Taxonomy
• Dynamic Time-Aware Attention to Speaker Roles and Contexts for Spoken Language Understanding
• ImageNet pre-trained models with batch normalization
• Recurrent Autoregressive Networks for Online Multi-Object Tracking
• Opinion Recommendation using Neural Memory Model
• Gated End-to-End Memory Networks
• Visual Text Correction
• ORTHOSEG: A DEEP MULTIMODAL CONVOLUTONAL NEURAL NETWORK ARCHITECTURE FOR SEMANTIC SEGMENTATION OF ORTHOIMAGERY
• SELC: Self-Supervised Efficient Local Correspondence Learning for Low Quality Images
• Visual Fashion-Product Search at SK Planet
• Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module
• Robust and Scalable Differentiable Neural Computer for Question Answering
• Hadamard product in deep learning: Introduction, Advances and Challenges
• Finding ReMO (Related Memory Object): A Simple Neural Architecture for Text based Reasoning

Discussions

• Dynamic Memory Networks for Visual and Textual Question Answering [hn, 122 points, 15 comments]

Related

• Visual Question Answering with Memory-Augmented Networks
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Textually Enriched Neural Module Networks for Visual Question Answering
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Ask Me Even More: Dynamic Memory Tensor Networks (Extended Model)