Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models

2025/06/03 by Orchid Chetia Phukan, Phukan, Orchid Chetia, Girish +12 · 1 citation
Computer Science · Medicine · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Multimedia (cs.MM) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Voice and Speech Disorders #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.03364

openalex publication_date 2025/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we introduce the task of singing voice deepfake source attribution (SVDSA). We hypothesize that multimodal foundation models (MMFMs) such as ImageBind, LanguageBind will be most effective for SVDSA as they are better equipped for capturing subtle source-specific characteristics-such as unique timbre, pitch manipulation, or synthesis artifacts of each singing voice deepfake source due to their cross-modality pre-training. Our experiments with MMFMs, speech foundation models and music foundation models verify the hypothesis that MMFMs are the most effective for SVDSA. Furthermore, inspired from related research, we also explore fusion of foundation models (FMs) for improved SVDSA. To this end, we propose a novel framework, COFFE which employs Chernoff Distance as novel loss function for effective fusion of FMs. Through COFFE with the symphony of MMFMs, we attain the topmost performance in comparison to all the individual FMs and baseline fusion methods.

Citations

• Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024
• SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge
• Singing Voice Graph Modeling for SingFake Detection
• Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfake
• LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
• SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge (CtrSVDD Track, Test Set)
• Distinguishing Neural Speech Synthesis Models Through Fingerprints in Speech Waveforms
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
• Scaling Speech Technology to 1,000+ Languages
• ImageBind: One Embedding Space To Bind Them All
• Robust Speech Recognition via Large-Scale Weak Supervision
• MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning
• Source Tracing: Detecting Voice Spoofing
• Attacker Attribution of Audio Deepfakes
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Cited by

• Curved Worlds, Clear Boundaries: Generalizing Speech Deepfake Detection using Hyperbolic and Spherical Geometry Spaces

Related

• Are Music Foundation Models Better at Singing Voice Deepfake Detection? Far-Better Fuse them with Speech Foundation Models
• Speech Foundation Model Ensembles for the Controlled Singing Voice Deepfake Detection (CtrSVDD) Challenge 2024
• SVDD 2024: The Inaugural Singing Voice Deepfake Detection Challenge
• SVDD Challenge 2024: A Singing Voice Deepfake Detection Challenge (CtrSVDD Track, Test Set)
• Not All Deepfakes Are Created Equal: Triaging Audio Forgeries for Robust Deepfake Singer Identification