MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances

2025/09/21 by Lee, Junhyeok, Wang, Helin, Guan, Yaohan +4
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2509.17143

Abstract

We introduce MaskVCT, a zero-shot voice conversion (VC) model that offers multi-factor controllability through multiple classifier-free guidances (CFGs). While previous VC models rely on a fixed conditioning scheme, MaskVCT integrates diverse conditions in a single model. To further enhance robustness and control, the model can leverage continuous or quantized linguistic features to enhance intellgibility and speaker similarity, and can use or omit pitch contour to control prosody. These choices allow users to seamlessly balance speaker identity, linguistic content, and prosodic factors in a zero-shot VC setting. Extensive experiments demonstrate that MaskVCT achieves the best target speaker and accent similarities while obtaining competitive word and character error rates compared to existing baselines. Audio samples are available at https://maskvct.github.io/.

Citations

• Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
• GenVC: Self-Supervised Zero-Shot Voice Conversion
• Sylber: Syllabic Embedding Representation of Speech from Raw Audio
• SyllableLM: Learning Coarse Semantic Units for Speech Language Models
• Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models
• The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidance
• Self-Supervised Speech Representations are More Phonetic than Semantic
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Amphion: An Open-Source Audio, Music and Speech Generation Toolkit
• VoiceLDM: Text-to-Speech with Environmental Context
• Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context
• Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation
• High-Fidelity Audio Compression with Improved RVQGAN
• Simple and Controllable Music Generation
• LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
• CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice
• SoundStorm: Efficient Parallel Audio Generation
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis
• FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion
• High Fidelity Neural Audio Compression
• Classifier-Free Diffusion Guidance
• MaskGIT: Masked Generative Image Transformer
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Hi-Fi Multi-Speaker English TTS Dataset
• Speech Resynthesis from Discrete Disentangled Self-Supervised Representations
• Attention Is All You Need
• Categorical Reparameterization with Gumbel-Softmax
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion
• Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy
• Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion
• NoiseVC: Towards High Quality Zero-Shot Voice Conversion
• GenVC: Self-Supervised Zero-Shot Voice Conversion