2020/02/23 by Hadi Abdi Khojasteh, Ebrahim Ansari, Khojasteh, Hadi Abdi +5
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #E.0 #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #H.3.3 #I.2.0 #I.2.10 #I.2.6 #I.2.7 #I.4.0 #I.4.10 #I.5.0 #I.7.0 #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2002.10016
openalex publication_date 2020/02/23 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
This paper considers the task of matching images and sentences by learning a visual-textual embedding space for cross-modal retrieval. Finding such a space is a challenging task since the features and representations of text and image are not comparable. In this work, we introduce an end-to-end deep multimodal convolutional-recurrent network for learning both vision and language representations simultaneously to infer image-text similarity. The model learns which pairs are a match (positive) and which ones are a mismatch (negative) using a hinge-based triplet ranking. To learn about the joint representations, we leverage our newly extracted collection of tweets from Twitter. The main characteristic of our dataset is that the images and tweets are not standardized the same as the benchmarks. Furthermore, there can be a higher semantic correlation between the pictures and tweets contrary to benchmarks in which the descriptions are well-organized. Experimental results on MS-COCO benchmark dataset show that our model outperforms certain methods presented previously and has competitive performance compared to the state-of-the-art. The code and dataset have been made available publicly.