2016/11/17 by Long Chen, Chen, Long, Hanwang Zhang +12 · 39 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Benchmark (surveying) #Cartography #Channel (broadcasting) #Closed captioning #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Context (archaeology) #Convolutional neural network #Domain Adaptation and Few-Shot Learning #Encoding (memory) #FOS: Computer and information sciences #Feature (linguistics) #Image (mathematics) #Layer (electronics) #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Sentence #Spatial contextual awareness #cs.CV
paper · pdf · doi:10.48550/arxiv.1611.05594
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2016/11/17 · arxiv created 2017/04/12 · arxiv updated 2017/04/13 · openalex created_date 2019/06/27 · openalex updated_date 2026/08/06
Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism --- a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods.