Learning Deep Structure-Preserving Image-Text Embeddings

2015/11/19 by Liwei Wang, Yin Li, Wang, Liwei +3 · 8 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Advanced Image and Video Retrieval Techniques

paper · pdf · doi:10.48550/arxiv.1511.06078

Abstract

This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a large margin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and text-to-image retrieval. Our method achieves new state-of-the-art results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet classification with deep convolutional neural networks

Cited by

• Mapping of Lesion Images to Somatic Mutations
• Reliable Cross-modal Alignment via Prototype Iterative Construction
• RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
• Face and voice cross-modal association with learning convex feature embedding
• Multispectral Deep Neural Networks for Pedestrian Detection
• Parameter-Efficient Single Collaborative Branch for Recommendation
• HUSE: Hierarchical Universal Semantic Embeddings
• T-EMDE: Sketching-based global similarity for cross-modal retrieval

Related

• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Learning Robust Visual-Semantic Embeddings
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval Framework
• Intra-Modal Constraint Loss For Image-Text Retrieval