Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion

2025/07/19 by Yu Zhang, Zhang, Yu, Zhiyao Duan +2
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.14534

openalex publication_date 2025/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Zero-shot online voice conversion (VC) holds significant promise for real-time communications and entertainment. However, current VC models struggle to preserve semantic fidelity under real-time constraints, deliver natural-sounding conversions, and adapt effectively to unseen speaker characteristics. To address these challenges, we introduce Conan, a chunkwise online zero-shot voice conversion model that preserves the content of the source while matching the voice timbre and styles of reference speech. Conan comprises three core components: 1) a Stream Content Extractor that leverages Emformer for low-latency streaming content encoding; 2) an Adaptive Style Encoder that extracts fine-grained stylistic features from reference speech for enhanced style adaptation; 3) a Causal Shuffle Vocoder that implements a fully causal HiFiGAN using a pixel-shuffle mechanism. Experimental evaluations demonstrate that Conan outperforms baseline models in subjective and objective metrics. Audio samples can be found at https://aaronz345.github.io/ConanDemo.

Citations

• TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis
• ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
• Versatile Framework for Song Generation with Prompt-based Control
• CSSinger: End-to-End Chunkwise Streaming Singing Voice Synthesis System Based on Conditional Variational Autoencoder
• TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control
• GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• End-to-end streaming model for low-latency speech anonymization
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion
• StreamVC: Real-Time Low-Latency Voice Conversion
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis
• DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion
• Online Clustered Codebook
• Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
• LM-VC: Zero-shot Voice Conversion via Speech Generation based on Language Models
• ALO-VC: Any-to-any Low-latency One-shot Voice Conversion
• Any-to-Any Generation via Composable Diffusion
• QuickVC: Any-to-many Voice Conversion Using Inverse Short-time Fourier Transform for Faster Conversion
• Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
• HuBERT: Self-Supervised Speech Representation Learning by Masked\n Prediction of Hidden Units
• Non-autoregressive sequence-to-sequence voice conversion
• FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion
• STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• FastPitch: Parallel Text-to-speech with Pitch Prediction
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
• AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss
• One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalization
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• Neural Discrete Representation Learning
• Attention Is All You Need
• Least Squares Generative Adversarial Networks
• Multi-level Temporal-channel Speaker Retrieval for Zero-shot Voice Conversion

Related

• SynthVC: Leveraging Synthetic Data for End-to-End Low Latency Streaming Voice Conversion
• StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion
• ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations
• Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling
• Pureformer-VC: Non-parallel Voice Conversion with Pure Stylized Transformer Blocks and Triplet Discriminative Training