Visual Question Answering with Memory-Augmented Networks

2017/07/17 by Chao Ma, Ma, Chao, Chunhua Shen +7
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1707.04968

openalex publication_date 2017/07/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we exploit a memory-augmented neural network to predict accurate answers to visual questions, even when those answers occur rarely in the training set. The memory network incorporates both internal and external memory blocks and selectively pays attention to each training exemplar. We show that memory-augmented neural networks are able to maintain a relatively long-term memory of scarce training exemplars, which is important for visual question answering due to the heavy-tailed distribution of answers in a general VQA setting. Experimental results on two large-scale benchmark datasets show the favorable performance of the proposed algorithm with a comparison to state of the art.

Citations

• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Long Short-Term Memory
• Bidirectional recurrent neural networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Deep Networks with Internal Selective Attention through Feedback\n Connections
• Simple Baseline for Visual Question Answering
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Effective Approaches to Attention-based Neural Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• VQA: Visual Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about\n Images
• Learning to Remember Rare Events
• Recurrent Neural Networks with External Memory for Language Understanding

Related

• Dynamic Memory Networks for Visual and Textual Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge
• Learning Visual Knowledge Memory Networks for Visual Question Answering