Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation

2024/07/07 by Haorui He, Zengqiang Shang, He, Haorui +25 · 131 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Speech Recognition and Synthesis #Speech and dialogue systems #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2407.05361

openalex publication_date 2024/07/07 · openalex created_date 2024/07/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in speech generation models have been significantly driven by the use of large-scale training data. However, producing highly spontaneous, human-like speech remains a challenge due to the scarcity of large, diverse, and spontaneous speech datasets. In response, we introduce Emilia, the first large-scale, multilingual, and diverse speech generation dataset. Emilia starts with over 101k hours of speech across six languages, covering a wide range of speaking styles to enable more natural and spontaneous speech generation. To facilitate the scale-up of Emilia, we also present Emilia-Pipe, the first open-source preprocessing pipeline designed to efficiently transform raw, in-the-wild speech data into high-quality training data with speech annotations. Experimental results demonstrate the effectiveness of both Emilia and Emilia-Pipe. Demos are available at: https://emilia-dataset.github.io/Emilia-Demo-Page/.

Cited by

• QuarkAudio Technical Report
• Aliasing-Free Neural Audio Synthesis
• YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases
• M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis
• Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
• Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
• Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding
• Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Reward
• Time-Layer Adaptive Alignment for Speaker Similarity in Flow-Matching Based Zero-Shot TTS
• The Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speech
• UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
• NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion
• UniTok-Audio: A Unified Audio Generation Framework via Generative Modeling on Discrete Codec Tokens
• SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level
• Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models
• TwinShift: Benchmarking Audio Deepfake Detection across Synthesizer and Speaker Shifts
• Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation
• OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
• UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
• Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluator
• InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
• Closing the Gap Between Text and Speech Understanding in LLMs
• DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation
• Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking
• SynthVC: Leveraging Synthetic Data for End-to-End Low Latency Streaming Voice Conversion
• MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows
• IntMeanFlow: Few-step Speech Generation with Integral Velocity Distillation
• Position: Towards Responsible Evaluation for Text-to-Speech
• TokenChain: A Discrete Speech Chain via Semantic Token Modeling
• UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
• Synthetic Audio Forensics Evaluation (SAFE) Challenge
• Evaluation of preprocessing pipelines in the creation of in-the-wild TTS datasets
• FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates
• EuroSpeech: A Multilingual Speech Corpus
• MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech
• SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement
• VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
• StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
• Semantic-VAE: Semantic-Alignment Latent Representation for Better Speech Synthesis
• AUV: Teaching Audio Universal Vector Quantization with Single Nested Codebook
• DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptation
• UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice
• Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration
• CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
• Selective Classifier-free Guidance for Zero-shot Text-to-speech
• VocalRender: Score-Native Singing Voice Synthesis for Real-World Composition
• Training Flow Matching Models with Reliable Labels via Self-Purification
• HarmoniFuse: A Component-Selective and Prompt-Adaptive Framework for Multi-Task Speech Language Modeling
• Nord-Parl-TTS: Finnish and Swedish TTS Dataset from Parliament Speech
• MBCodec:Thorough disentangle for high-fidelity audio compression
• MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech
• UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching
• VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency
• The Singing Voice Conversion Challenge 2025: From Singer Identity Conversion To Singing Style Conversion
• TISDiSS: A Training-Time and Inference-Time Scalable Framework for Discriminative Source Separation
• Speech Language Models for Under-Represented Languages: Insights from Wolof
• Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
• Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis
• Multi-Metric Preference Alignment for Generative Speech Restoration
• DiFlow-TTS: Discrete Flow Matching with Factorized Speech Tokens for Low-Latency Zero-Shot Text-To-Speech
• Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates
• Over-the-Air Adversarial Attack Detection: from Datasets to Defenses
• Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
• LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis
• Flavors of Moonshine: Tiny Specialized ASR Models for Edge Devices
• AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation
• SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
• Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generation
• Robust Residual Finite Scalar Quantization for Neural Compression
• Integrating Feedback Loss from Bi-modal Sarcasm Detector for Sarcastic Speech Synthesis
• Cryfish: On deep audio analysis with Large Language Models
• Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation
• OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue
• Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling
• UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
• A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding
• REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers
• Fairness in Dysarthric Speech Synthesis: Understanding Intrinsic Bias in Dysarthric Speech Cloning using F5-TTS
• Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech
• MiDashengLM: Efficient Audio Understanding with General Audio Captions
• SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
• Whilter: A Whisper-based Data Filter for "In-the-Wild" Speech Corpora Using Utterance-level Multi-Task Classification
• HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
• The TEA-ASLP System for Multilingual Conversational Speech Recognition and Speech Diarization in MLC-SLM 2025 Challenge
• DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
• NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech
• ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching
• ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark
• OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model
• TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
• Traceable TTS: Toward Watermark-Free TTS with Strong Traceability
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
• StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decoding
• XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs
• TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• OpusLM: A Family of Open Unified Speech Language Models
• LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization
• ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling
• MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
• Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment
• HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset
• CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech
• VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
• Leveraging Large Language Models for Sarcastic Speech Annotation in Sarcasm Detection
• Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages
• VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation
• Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling
• UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information
• From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition
• MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing
• Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey
• Neurodyne: Neural Pitch Manipulation with Representation Learning and Cycle-Consistency GAN
• A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model
• Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation
• DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation
• OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching
• SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset
• DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis
• Teochew-Wild: The First In-the-wild Teochew Dataset with Orthographic Annotations
• Voice Cloning: Comprehensive Survey
• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
• GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model
• CLASVS: Continuous-Latent Autoregression for Melody-Preserving Lyric Editing in Singing Voice Synthesis
• Kimi-Audio Technical Report
• Solid State Bus-Comp: A Large-Scale and Diverse Dataset for Dynamic Range Compressor Virtual Analog Modeling
• AffectDF: The Most Comprehensive Benchmark for Speech Deepfake Detection against Emotionally Expressive Attacks
• Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation
• Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis

Related

• Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation
• SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models
• FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
• Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development
• HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset