Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

2017/11/17 by Jiuxiang Gu, Jianfei Cai, Gu, Jiuxiang +7 · 5 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1711.06420

openalex publication_date 2017/11/17 · openalex created_date 2017/12/04 · openalex updated_date 2026/07/28

Abstract

Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.

Citations

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Recent advances in convolutional neural networks

Cited by

• PHyCLIP: ℓ₁-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
• Multimodal Fine-grained Reasoning for Post Quality Evaluation
• HUSE: Hierarchical Universal Semantic Embeddings
• Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval

Related

• Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
• Revisiting Cross Modal Retrieval
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs
• Towards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features