XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

2021/11/17 by Arun Babu, Changhan Wang, Babu, Arun +23 · 107 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2111.09296

openalex publication_date 2021/11/17 · openalex created_date 2022/05/05 · openalex updated_date 2026/07/28

Abstract

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.

Cited by

• Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognition
• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
• Leveraging Gradient Reversal Loss and Multitask Learning for Datasets-Aware Audio Deepfake Detection
• Improving Zero-Shot Phonetic Classification through Language-Agnostic Articulatory Features
• Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
• MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
• A Data-Centric Approach to Generalizable Speech Deepfake Detection
• What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels
• Always Keep Your Promises: A Model-Agnostic Attribution Algorithm for Neural Networks
• Dialect Identification Using Resource-Efficient Fine-Tuning Approaches
• Benchmarking Automatic Speech Recognition Models for African Languages
• Scaling HuBERT for African Languages: From Base to Large and XL
• Zero-Shot Context-Aware ASR for Diverse Arabic Varieties
• Investigating self-supervised representations for audio-visual deepfake detection
• On the Cross-lingual Transferability of Pre-trained wav2vec2-based Models
• How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer
• Towards Attribution of Generators and Emotional Manipulation in Cross-Lingual Synthetic Speech using Geometric Learning
• TEDxTN: A Three-way Speech Translation Corpus for Code-Switched Tunisian Arabic - English
• ADI-20: Arabic Dialect Identification dataset and models
• Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
• National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solution
• How to Evaluate Speech Translation with Source-Aware Neural MT Metrics
• CARE: A Multimodal Corpus for Studying Speech and Non-Verbal Communication Across Multiple Medical Conditions
• ThinkOmni: A Reasoning-Driven Omni-Modal LLM Framework for Audio Forgery Detection and Localization
• POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
• BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptation
• The Tonogenesis Continuum in Tibetan: A Computational Investigation
• AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch
• SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation
• Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects
• XLSR-Kanformer: A KAN-Intergrated model for Synthetic Speech Detection
• WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection
• Evaluating Self-Supervised Speech Models via Text-Based LLMS
• Forensic Similarity for Speech Deepfakes
• SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentation
• Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives
• HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech
• AUDDT: Audio Unified Deepfake Detection Benchmark Toolkit
• Weakly Supervised Phonological Features for Pathological Speech Analysis
• Teffic-Audio: Tell Fact from Fiction
• FasterVoiceGrad: Faster One-step Diffusion-Based Voice Conversion with Adversarial Diffusion Conversion Distillation
• MoRoVoc: A Large Dataset for Geographical Variation Identification of the Spoken Romanian Language
• Rethinking Cross-Corpus Speech Emotion Recognition Benchmarking: Are Paralinguistic Pre-Trained Representations Sufficient?
• Investigating Polyglot Speech Foundation Models for Learning Collective Emotion from Crowds
• Speech Language Models for Under-Represented Languages: Insights from Wolof
• Towards Building Speech Large Language Models for Multitask Understanding in Low-Resource Languages
• HARNESS: Lightweight Distilled Arabic Speech Foundation Models
• Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
• Geolocation-Aware Robust Spoken Language Identification
• Improving Out-of-Domain Audio Deepfake Detection via Layer Selection and Fusion of SSL-Based Countermeasures
• In-domain SSL pre-training and streaming ASR
• SENSE models: an open source solution for multilingual and multimodal semantic-based tasks
• Towards Data Drift Monitoring for Speech Deepfake Detection in the context of MLOps
• UniCoM: A Universal Code-Switching Speech Generator
• The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
• AUDETER: A Large-scale Dataset for Deepfake Audio Detection in Open Worlds
• An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
• English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM
• Generalizable Audio Spoofing Detection using Non-Semantic Representations
• Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System
• CAMÕES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese
• CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing
• Arabic ASR on the SADA Large-Scale Arabic Speech Corpus with Transformer-Based Models
• Cryfish: On deep audio analysis with Large Language Models
• Emphasis Sensitivity in Speech Representations
• Fake Speech Wild: Detecting Deepfake Speech on Social Media Platform
• Exploring Cross-Utterance Speech Contexts for Conformer-Transducer Speech Recognition Systems
• Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative
• Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
• SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysis
• Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0
• Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech
• LENS-DF: Deepfake Detection and Temporal Localization for Long-Form Noisy Speech
• Generalizable Audio Deepfake Detection via Hierarchical Structure Learning and Feature Whitening in Poincaré sphere
• Fusion of Modulation Spectrogram and SSL with Multi-head Attention for Fake Speech Detection
• SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods
• A Deep Learning Automatic Speech Recognition Model for Shona Language
• Frame-level Temporal Difference Learning for Partial Deepfake Speech Detection
• Enhancing In-Domain and Out-Domain EmoFake Detection via Cooperative Multilingual Speech Foundation Models
• Advancing STT for Low-Resource Real-World Speech
• mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks
• Towards Scalable AASIST: Refining Graph Attention for Speech Deepfake Detection
• On Barriers to Archival Audio Processing
• Mask6D: Masked Pose Priors For 6D Object Pose Estimation
• Word stress in self-supervised speech models: A cross-linguistic comparison
• Self-supervised learning of speech representations with Dutch archival data
• Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation
• Benchmarking Akan ASR Models Across Domain-Specific Datasets: A Comparative Evaluation of Performance, Scalability, and Adaptability
• Mitigating Language Mismatch in SSL-Based Speaker Anonymization
• Dataset to publication: "Neighbors and relatives: How do speech embeddings reflect linguistic connections across the world?"
• Cross-lingual Data Selection Using Clip-level Acoustic Similarity for Enhancing Low-resource Automatic Speech Recognition
• HighRateMOS: Sampling-Rate Aware Modeling for Speech Quality Assessment
• Post-training for Deepfake Speech Detection
• DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
• Low-resource keyword spotting using contrastively trained transformer acoustic word embeddings
• Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning
• A novel fusion architecture for detecting Parkinson’s Disease using semi-supervised speech embeddings
• Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages
• Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models
• Towards Generalized Source Tracing for Codec-Based Deepfake Speech
• From Sharpness to Better Generalization for Speech Deepfake Detection
• Joint ASR and Speaker Role Tagging with Serialized Output Training
• SynHate: Detecting Hate Speech in Synthetic Deepfake Audio
• A Practitioner's Guide to Building ASR Models for Low-Resource Languages: A Case Study on Scottish Gaelic
• ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition

Related

• Unsupervised Cross-lingual Representation Learning for Speech\n Recognition
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• Towards Robust Speech Representation Learning for Thousands of Languages
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Improved Language Identification Through Cross-Lingual Self-Supervised Learning