Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

2023/03/07 by Ziqiang Zhang, Long Zhou, Zhang, Ziqiang +23 · 22 citations
Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2303.03926

openalex publication_date 2023/03/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker's voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at \urlhttps://aka.ms/vallex.

Cited by

• Training Text-to-Speech Model with Purely Synthetic Data: Feasibility, Sensitivity, and Generalization Capability
• Harmonic-Percussive Disentangled Neural Audio Codec for Bandwidth Extension
• InstructAudio: Unified speech and music generation with natural language instruction
• E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
• Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluator
• U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation
• UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
• Universal Discrete-Domain Speech Enhancement
• How I Built ASR for Endangered Languages with a Spoken Dictionary
• Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba
• Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
• Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling
• CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance
• Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters
• Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis
• CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
• Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets
• XEmoRAG: Cross-Lingual Emotion Transfer with Controllable Intensity Using Retrieval-Augmented Generation
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec
• Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
• Next Tokens Denoising for Speech Synthesis
• Data Augmentation for Spoken Grammatical Error Correction

Related

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning
• Zero-Shot Emotion Transfer For Cross-Lingual Speech Synthesis
• VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing
• Efficient neural speech synthesis for low-resource languages through\n multilingual modeling