Collecting, Curating, and Annotating Good Quality Speech deepfake dataset for Famous Figures: Process and Challenges

2025/06/30 by Ali, Hashim, Subramani, Surya, Varahamurthy, Raksha +3 · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2507.00324

Abstract

Recent advances in speech synthesis have introduced unprecedented challenges in maintaining voice authenticity, particularly concerning public figures who are frequent targets of impersonation attacks. This paper presents a comprehensive methodology for collecting, curating, and generating synthetic speech data for political figures and a detailed analysis of challenges encountered. We introduce a systematic approach incorporating an automated pipeline for collecting high-quality bonafide speech samples, featuring transcription-based segmentation that significantly improves synthetic speech quality. We experimented with various synthesis approaches; from single-speaker to zero-shot synthesis, and documented the evolution of our methodology. The resulting dataset comprises bonafide and synthetic speech samples from ten public figures, demonstrating superior quality with a NISQA-TTS naturalness score of 3.69 and the highest human misclassification rate of 61.9%.

Citations

• Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• SpoofCeleb: Speech Deepfake Detection and SASV In The Wild
• DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Dataset
• SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale
• E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
• Is Audio Spoof Detection Robust to Laundering Attacks?
• XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model
• The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio
• MLAAD: The Multi-Language Audio Anti-Spoofing Dataset
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Does Audio Deepfake Detection Generalize?
• ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
• A Survey on Neural Speech Synthesis
• UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Voxceleb: Large-scale speaker verification in the wild
• ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech
• Tacotron: Towards End-to-End Speech Synthesis

Cited by

• Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge System

Related

• Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis
• A Comprehensive Survey with Critical Analysis for Deepfake Speech Detection
• Can we steal your vocal identity from the Internet?: Initial\n investigation of cloning Obama's voice using GAN, WaveNet and low-quality\n found data
• Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion
• Speaker Privacy and Security in the Big Data Era: Protection and Defense against Deepfake