mHuBERT-147: A Compact Multilingual HuBERT Model

2024/06/10 by Marcely Zanon Boito, Boito, Marcely Zanon, Vivek Iyer +7 · 18 citations
Computer Science · #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2406.06371

Abstract

We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-based clustering, achieving 5.2x faster label assignment than the original method. We also apply a new multilingual batching up-sampling strategy, leveraging both language and dataset diversity. After 3 training iterations, our compact 95M parameter mHuBERT-147 outperforms larger models trained on substantially more data. We rank second and first on the ML-SUPERB 10min and 1h leaderboards, with SOTA scores for 3 tasks. Across ASR/LID tasks, our model consistently surpasses XLS-R (300M params; 436K hours) and demonstrates strong competitiveness against the much larger MMS (1B params; 491K hours). Our findings indicate that mHuBERT-147 is a promising model for multilingual speech tasks, offering an unprecedented balance between high performance and parameter efficiency.

Cited by

• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
• A Cross-lingual Comparison of Human and Classification Model Entrainment Behavior in Code-switched Speech Settings
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
• Leveraging Language Information for Target Language Extraction
• How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer
• National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solution
• MT-HuBERT: Self-Supervised Mix-Training for Few-Shot Keyword Spotting in Mixed Speech
• Linguistically Informed Evaluation of Multilingual ASR for African Languages
• DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching
• Unsupervised lexicon learning from speech is limited by representations rather than clustering
• Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects
• DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment
• Speech Language Models for Under-Represented Languages: Insights from Wolof
• BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings
• UniVerse-1: Unified Audio-Video Generation via Stitching of Experts
• Layer-wise Analysis for Quality of Multilingual Synthesized Speech
• Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification

Related

• Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond
• Scaling HuBERT for African Languages: From Base to Large and XL
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
• Exploration on HuBERT with Multiple Resolutions
• Scaling End-to-End Models for Large-Scale Multilingual ASR