M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval

2022/11/02 by Berry, Layne, Shih, Yi-Jen, Wang, Hsuan-Fu +3 · 2 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2211.01180

Abstract

This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval. For non-English image-speech retrieval, we outperform the current state-of-the-art performance by a wide margin both when training separate models for each language, and with a single model which processes speech in all three languages. We identify key differences in model behavior and performance between English and non-English settings, attributable to the English-only pre-training of CLIP and HuBERT, and investigate how fine-tuning the pre-trained models impacts these differences. Finally, we show that our models can be used for mono- and cross-lingual speech-text retrieval and cross-lingual speech-speech retrieval, despite never having seen any parallel speech-text or speech-speech data during training.

Cited by

• Unifying Model and Layer Fusion for Speech Foundation Models
• Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models

Related

• Towards Zero-shot Cross-lingual Image Retrieval
• SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
• Towards Zero-shot Cross-lingual Image Retrieval and Tagging
• Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning
• uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data