Instance-aware Image and Sentence Matching with Selective Multimodal LSTM

2016/11/17 by Yan Huang, Wei Wang, Huang, Yan +3 · 8 citations
Computer Science · Mathematics · #Advanced Image and Video Retrieval Techniques #Artificial intelligence #Benchmark (surveying) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Context (archaeology) #Data mining #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Image (mathematics) #Matching (statistics) #Mathematics #Measure (data warehouse) #Multimodal Machine Learning Applications #Natural language processing #Pairwise comparison #Pattern recognition (psychology) #Sentence #Similarity (geometry) #Statistics #cs.CV

paper · pdf · doi:10.48550/arxiv.1611.05588

published in arXiv (Cornell University) (Cornell University)

arxiv created 2016/11/17 · openalex publication_date 2016/11/17 · arxiv updated 2017/12/07 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local similarities between pairwise instances of image (objects) and sentence (words), we propose a selective multimodal Long Short-Term Memory network (sm-LSTM) for instance-aware image and sentence matching. The sm-LSTM includes a multimodal context-modulated attention scheme at each timestep that can selectively attend to a pair of instances of image and sentence, by predicting pairwise instance-aware saliency maps for image and sentence. For selected pairwise instances, their representations are obtained based on the predicted saliency maps, and then compared to measure their local similarity. By similarly measuring multiple local similarities within a few timesteps, the sm-LSTM sequentially aggregates them with hidden states to obtain a final matching score as the desired global similarity. Extensive experiments show that our model can well match image and sentence with complex content, and achieve the state-of-the-art results on two public benchmark datasets.

Citations

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show and tell: A neural image caption generator
• Long Short-Term Memory
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Bidirectional recurrent neural networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Explain Images with Multimodal Recurrent Neural Networks
• Contextual LSTM (CLSTM) models for Large scale NLP tasks
• Multimodal Convolutional Neural Networks for Matching Image and Sentence

Cited by

• OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment
• PREMISE: Matching-based Prediction for Accurate Review Recommendation

Related

• Saliency-Guided Attention Network for Image-Sentence Matching
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• phi-LSTM: A Phrase-based Hierarchical LSTM Model for Image Captioning
• Learning Semantic Concepts and Order for Image and Sentence Matching
• Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching