SyncVoice: Towards Video Dubbing with Vision-Augmented Pretrained TTS Model

2025/11/23 by Kaidi Wang, Wang, Kaidi, Yi He +19
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Sound (cs.SD) #Subtitles and Audiovisual Media #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2512.05126

openalex publication_date 2025/11/23 · openalex created_date 2025/12/09 · openalex updated_date 2026/07/28

Abstract

Video dubbing aims to generate high-fidelity speech that is precisely temporally aligned with the visual content. Existing methods still suffer from limitations in speech naturalness and audio-visual synchronization, and are limited to monolingual settings. To address these challenges, we propose SyncVoice, a vision-augmented video dubbing framework built upon a pretrained text-to-speech (TTS) model. By fine-tuning the TTS model on audio-visual data, we achieve strong audiovisual consistency. We propose a Dual Speaker Encoder to effectively mitigate inter-language interference in cross-lingual speech synthesis and explore the application of video dubbing in video translation scenarios. Experimental results show that SyncVoice achieves high-fidelity speech generation with strong synchronization performance, demonstrating its potential in video dubbing tasks.

Citations

• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
• MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder
• AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
• VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models
• Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing
• EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing
• LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing
• CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
• Learning to Dub Movies via Hierarchical Prosody Models
• VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
• Classifier-Free Diffusion Guidance
• Neural Dubber: Dubbing for Videos According to Scripts
• VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over
• Lipreading using Temporal Convolutional Networks

Related

• Large-scale multilingual audio visual dubbing
• VSpeechLM: A Visual Speech Language Model for Visual Text-to-Speech Task
• More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech
• Video Editing for Audio-Visual Dubbing
• Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation