Language translation, and change of accent for speech-to-speech task using diffusion model

2025/05/04 by Abhishek Mishra, Ritesh Sur Chowdhury, Mishra, Abhishek +6
Computer Science · Medicine · Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Phonetics and Phonology Research #Speech Recognition and Synthesis #Voice and Speech Disorders

paper · pdf · doi:10.48550/arxiv.2505.04639

openalex publication_date 2025/05/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Speech-to-speech translation (S2ST) aims to convert spoken input in one language to spoken output in another, typically focusing on either language translation or accent adaptation. However, effective cross-cultural communication requires handling both aspects simultaneously - translating content while adapting the speaker's accent to match the target language context. In this work, we propose a unified approach for simultaneous speech translation and change of accent, a task that remains underexplored in current literature. Our method reformulates the problem as a conditional generation task, where target speech is generated based on phonemes and guided by target speech features. Leveraging the power of diffusion models, known for high-fidelity generative capabilities, we adapt text-to-image diffusion strategies by conditioning on source speech transcriptions and generating Mel spectrograms representing the target speech with desired linguistic and accentual attributes. This integrated framework enables joint optimization of translation and accent adaptation, offering a more parameter-efficient and effective model compared to traditional pipelines.

Citations

• SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
• Adding Conditional Control to Text-to-Image Diffusion Models
• Adding Conditional Control to Text-to-Image Diffusion Models
• High Fidelity Neural Audio Compression
• Flow Matching for Generative Modeling
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• Improved Denoising Diffusion Probabilistic Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• Beyond English-Centric Multilingual Machine Translation
• Denoising Diffusion Implicit Models
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Multilingual Denoising Pre-training for Neural Machine Translation
• Direct speech-to-speech translation with a sequence-to-sequence model

Related

• Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing
• Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation
• Direct Speech to Speech Translation: A Review
• S2ST-Omni: An Efficient Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Progressive Fine-tuning
• Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer