Multimodal One-Shot Learning of Speech and Images

2018/11/09 by Ryan Eloff, Eloff, Ryan, Herman A. Engelbrecht +3
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #cs.CL #cs.CV #cs.LG #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1811.03875

5 pages, 1 figure, 3 tables; accepted to ICASSP 2019

arxiv created 2019/04/15 · arxiv updated 2019/04/16

Abstract

Imagine a robot is shown new concepts visually together with spoken tags, e.g. "milk", "eggs", "butter". After seeing one paired audio-visual example per class, it is shown a new set of unseen instances of these objects, and asked to pick the "milk". Without receiving any hard labels, could it learn to match the new continuous speech input to the correct visual instance? Although unimodal one-shot learning has been studied, where one labelled example in a single modality is given per class, this example motivates multimodal one-shot learning. Our main contribution is to formally define this task, and to propose several baseline and advanced models. We use a dataset of paired spoken and visual digits to specifically investigate recent advances in Siamese convolutional neural networks. Our best Siamese model achieves twice the accuracy of a nearest neighbour model using pixel-distance over images and dynamic time warping over speech in 11-way cross-modal matching.

Related

• Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images
• Direct multimodal few-shot learning of speech and images
• Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
• Multimodal Few-Shot Learning with Frozen Language Models
• Visually grounded learning of keyword prediction from untranscribed speech