XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

2021/11/17 by Arun Babu, Changhan Wang, Babu, Arun +23 · 152 citations
Computer Science · Engineering · #Artificial intelligence #Audio and Speech Processing (eess.AS) #Benchmark (surveying) #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #FOS: Electrical engineering #Intermediate language #Machine translation #Music and Audio Processing #Natural Language Processing Techniques #Natural language processing #Parallel corpora #Range (aeronautics) #Representation (politics) #Scale (ratio) #Sound (cs.SD) #Speech Recognition and Synthesis #Speech recognition #Word error rate #cs.CL #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2111.09296

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/11/17 · arxiv created 2021/12/16 · arxiv updated 2021/12/17 · openalex created_date 2022/05/05 · openalex updated_date 2026/08/05

Abstract

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.

Cited by

• Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognition
• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
• Leveraging Gradient Reversal Loss and Multitask Learning for Datasets-Aware Audio Deepfake Detection
• Improving Zero-Shot Phonetic Classification through Language-Agnostic Articulatory Features
• Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study
• SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
• MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
• A Data-Centric Approach to Generalizable Speech Deepfake Detection
• What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels
• Always Keep Your Promises: A Model-Agnostic Attribution Algorithm for Neural Networks
• Dialect Identification Using Resource-Efficient Fine-Tuning Approaches
• Benchmarking Automatic Speech Recognition Models for African Languages
• Scaling HuBERT for African Languages: From Base to Large and XL
• Zero-Shot Context-Aware ASR for Diverse Arabic Varieties
• Investigating self-supervised representations for audio-visual deepfake detection
• On the Cross-lingual Transferability of Pre-trained wav2vec2-based Models
• How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer
• Towards Attribution of Generators and Emotional Manipulation in Cross-Lingual Synthetic Speech using Geometric Learning
• TEDxTN: A Three-way Speech Translation Corpus for Code-Switched Tunisian Arabic - English
• ADI-20: Arabic Dialect Identification dataset and models
• Towards Effective and Efficient Non-autoregressive decoders for Conformer and LLM-based ASR using Block-based Attention Mask
• National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solution
• How to Evaluate Speech Translation with Source-Aware Neural MT Metrics
• CARE: A Multimodal Corpus for Studying Speech and Non-Verbal Communication Across Multiple Medical Conditions
• ThinkOmni: A Reasoning-Driven Omni-Modal LLM Framework for Audio Forgery Detection and Localization
• POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
• BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptation
• The Tonogenesis Continuum in Tibetan: A Computational Investigation
• AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratch
• SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation
• Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects
• XLSR-Kanformer: A KAN-Intergrated model for Synthetic Speech Detection
• WaveSP-Net: Learnable Wavelet-Domain Sparse Prompt Tuning for Speech Deepfake Detection
• Evaluating Self-Supervised Speech Models via Text-Based LLMS
• Forensic Similarity for Speech Deepfakes
• SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentation
• Code-switching Speech Recognition Under the Lens: Model- and Data-Centric Perspectives
• HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech
• AUDDT: Audio Unified Deepfake Detection Benchmark Toolkit
• Weakly Supervised Phonological Features for Pathological Speech Analysis
• Teffic-Audio: Tell Fact from Fiction
• FasterVoiceGrad: Faster One-step Diffusion-Based Voice Conversion with Adversarial Diffusion Conversion Distillation
• MoRoVoc: A Large Dataset for Geographical Variation Identification of the Spoken Romanian Language
• Rethinking Cross-Corpus Speech Emotion Recognition Benchmarking: Are Paralinguistic Pre-Trained Representations Sufficient?
• Investigating Polyglot Speech Foundation Models for Learning Collective Emotion from Crowds
• Speech Language Models for Under-Represented Languages: Insights from Wolof
• Towards Building Speech Large Language Models for Multitask Understanding in Low-Resource Languages
• HARNESS: Lightweight Distilled Arabic Speech Foundation Models
• Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST
• Geolocation-Aware Robust Spoken Language Identification
• Improving Out-of-Domain Audio Deepfake Detection via Layer Selection and Fusion of SSL-Based Countermeasures
• In-domain SSL pre-training and streaming ASR
• SENSE models: an open source solution for multilingual and multimodal semantic-based tasks
• Towards Data Drift Monitoring for Speech Deepfake Detection in the context of MLOps
• UniCoM: A Universal Code-Switching Speech Generator
• The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
• AUDETER: A Large-scale Dataset for Deepfake Audio Detection in Open Worlds
• An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
• English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM
• Generalizable Audio Spoofing Detection using Non-Semantic Representations
• Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System
• CAMÕES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese
• CUPE: Contextless Universal Phoneme Encoder for Language-Agnostic Speech Processing
• Arabic ASR on the SADA Large-Scale Arabic Speech Corpus with Transformer-Based Models
• Cryfish: On deep audio analysis with Large Language Models
• Emphasis Sensitivity in Speech Representations
• Fake Speech Wild: Detecting Deepfake Speech on Social Media Platform
• Exploring Cross-Utterance Speech Contexts for Conformer-Transducer Speech Recognition Systems
• Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative
• Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
• SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysis
• Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0
• Balalaika: Data-Centric, Prosody-Aware Annotation Pipeline for Russian Speech
• LENS-DF: Deepfake Detection and Temporal Localization for Long-Form Noisy Speech
• Generalizable Audio Deepfake Detection via Hierarchical Structure Learning and Feature Whitening in Poincaré sphere
• Fusion of Modulation Spectrogram and SSL with Multi-head Attention for Fake Speech Detection
• SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Incorporating Cutting-Edge Generation Methods
• A Deep Learning Automatic Speech Recognition Model for Shona Language
• Frame-level Temporal Difference Learning for Partial Deepfake Speech Detection
• Enhancing In-Domain and Out-Domain EmoFake Detection via Cooperative Multilingual Speech Foundation Models
• Advancing STT for Low-Resource Real-World Speech
• mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks
• Towards Scalable AASIST: Refining Graph Attention for Speech Deepfake Detection
• On Barriers to Archival Audio Processing
• Mask6D: Masked Pose Priors For 6D Object Pose Estimation
• Word stress in self-supervised speech models: A cross-linguistic comparison
• Self-supervised learning of speech representations with Dutch archival data
• Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation
• Benchmarking Akan ASR Models Across Domain-Specific Datasets: A Comparative Evaluation of Performance, Scalability, and Adaptability
• Mitigating Language Mismatch in SSL-Based Speaker Anonymization
• Dataset to publication: "Neighbors and relatives: How do speech embeddings reflect linguistic connections across the world?"
• Cross-lingual Data Selection Using Clip-level Acoustic Similarity for Enhancing Low-resource Automatic Speech Recognition
• HighRateMOS: Sampling-Rate Aware Modeling for Speech Quality Assessment
• Post-training for Deepfake Speech Detection
• DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
• Low-resource keyword spotting using contrastively trained transformer acoustic word embeddings
• Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning
• A novel fusion architecture for detecting Parkinson’s Disease using semi-supervised speech embeddings
• Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages
• Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models
• Towards Generalized Source Tracing for Codec-Based Deepfake Speech
• From Sharpness to Better Generalization for Speech Deepfake Detection
• Joint ASR and Speaker Role Tagging with Serialized Output Training
• SynHate: Detecting Hate Speech in Synthetic Deepfake Audio
• A Practitioner's Guide to Building ASR Models for Low-Resource Languages: A Case Study on Scottish Gaelic
• ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
• HYFuse: Aligning Heterogeneous Speech Pre-Trained Representations in Hyperbolic Space for Speech Emotion Recognition
• A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation
• Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models
• PartialEdit: Identifying Partial Deepfakes in the Era of Neural Speech Editing
• Investigating the Reasonable Effectiveness of Speaker Pre-Trained Models and their Synergistic Power for SingMOS Prediction
• Evaluating Logit-Based GOP Scores for Mispronunciation Detection
• Towards Machine Unlearning for Paralinguistic Speech Processing
• Source Tracing of Synthetic Speech Systems Through Paralinguistic Pre-Trained Representations
• Mispronunciation Detection Without L2 Pronunciation Dataset in Low-Resource Setting: A Case Study in Finland Swedish
• GigaAM: Efficient Self-Supervised Learner for Speech Recognition
• Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
• Improving ASR Fairness for Cleft Lip and Palate Speech: A Study on Severity-Aware Data Mixing
• RPRA-ADD: Forgery Trace Enhancement-Driven Audio Deepfake Detection
• Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
• Voice Conversion Improves Cross-Domain Robustness for Spoken Arabic Dialect Identification
• When Humans Growl and Birds Speak: High-Fidelity Voice Conversion from Human to Animal and Designed Sounds
• Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC
• Few-Shot Speech Deepfake Detection Adaptation with Gaussian Processes
• ZIPA: A family of efficient models for multilingual phone recognition
• REWIND: Speech Time Reversal for Enhancing Speaker Representations in Diffusion-based Voice Conversion
• PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems
• Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead
• "KAN you hear me?" Exploring Kolmogorov-Arnold Networks for Spoken Language Understanding
• Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play
• EnvSDD: Benchmarking Environmental Sound Deepfake Detection
• Voice of a Continent: Mapping Africa's Speech Technology Frontier
• VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining
• Multi-Backbone Self-Supervised Ensembles for Audio Deepfake Detection and a Cross-Track Analysis of Generation-Detection Asymmetry
• From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition
• "Alexa, can you forget me?" Machine Unlearning Benchmark in Spoken Language Understanding
• On the Relevance of Clinical Assessment Tasks for the Automatic Detection of Parkinson's Disease Medication State from Speech
• ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality
• SHEET: A Multi-purpose Open-source Speech Human Evaluation Estimation Toolkit
• Analysis of ABC Frontend Audio Systems for the NIST-SRE24
• Naturalness-Aware Curriculum Learning with Dynamic Temperature for Speech Deepfake Detection
• Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings
• Hidden-Domain Routing for All-Type Audio Deepfake Detection
• Language translation, and change of accent for speech-to-speech task using diffusion model
• Weakly-supervised Audio Temporal Forgery Localization via Progressive Audio-language Co-learning Network
• Voice Cloning: Comprehensive Survey
• Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs
• Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models
• Where Do Self-Supervised Speech Models Become Unfair?
• Spatial Speech Translation: Translating Across Space With Binaural Hearables
• Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception

Related

• Unsupervised Cross-lingual Representation Learning for Speech Recognition
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• Towards Robust Speech Representation Learning for Thousands of Languages
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Improved Language Identification Through Cross-Lingual Self-Supervised Learning