REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

2025/08/07 by Jiang, Yuepeng, Ning, Ziqian, Wang, Shuai +5 · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2508.04996

Abstract

In real-world voice conversion applications, environmental noise in source speech and user demands for expressive output pose critical challenges. Traditional ASR-based methods ensure noise robustness but suppress prosody richness, while SSL-based models improve expressiveness but suffer from timbre leakage and noise sensitivity. This paper proposes REF-VC, a noise-robust expressive voice conversion system. Key innovations include: (1) A random erasing strategy to mitigate the information redundancy inherent in SSL features, enhancing noise robustness and expressiveness; (2) Implicit alignment inspired by E2TTS to suppress non-essential feature reconstruction; (3) Integration of Shortcut Models to accelerate flow matching inference, significantly reducing to 4 steps. Experimental results demonstrate that REF-VC outperforms baselines such as Seed-VC in zero-shot scenarios on the noisy set, while also performing comparably to Seed-VC on the clean set. In addition, REF-VC can be compatible with singing voice conversion within one model.

Citations

• Zero-shot Voice Conversion with Diffusion Transformers
• One Step Diffusion via Shortcut Models
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
• Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy
• SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
• VITS-Based Singing Voice Conversion Leveraging Whisper and multi-scale F0 Modeling
• DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Coding
• Self-Supervised Representations for Singing Voice Conversion
• Scalable Diffusion Models with Transformers
• FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
• Flow Matching for Generative Modeling
• WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit
• Noise-robust voice conversion with domain adversarial training
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Toward Degradation-Robust Voice Conversion
• WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations
• Accent and Speaker Disentanglement in Many-to-many Voice Conversion
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling

Cited by

• R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversion

Related

• R2-SVC: Towards Real-World Robust and Expressive Zero-shot Singing Voice Conversion
• ReFlow-VC: Zero-shot Voice Conversion Based on Rectified Flow and Speaker Feature Optimization
• Zero-shot Voice Conversion with Diffusion Transformers
• HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios
• StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching