Length Aware Speech Translation for Video Dubbing

2025/05/31 by Harveen Singh Chadha, Aswin Shanmugam Subramanian, Chadha, Harveen Singh +11 · 1 citation
Computer Science · #Speech and Audio Processing #Speech Recognition and Synthesis #Advanced Data Compression Techniques

paper · pdf · doi:10.48550/arxiv.2506.00740

Abstract

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.

Citations

• Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning
• VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
• Revisiting End-to-End Speech-to-Text Translation From Scratch
• FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
• Large-Scale Streaming End-to-End Speech Translation with Neural Transducers
• Isochrony-Aware Neural Machine Translation for Automatic Dubbing
• Machine Translation Verbosity Control for Automatic Dubbing
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Is 42 the Answer to Everything in Subtitling-oriented Speech\n Translation?
• Adapting End-to-End Speech Recognition for Readable Subtitles
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation

Cited by

• GlobalizeEd: A Multimodal Translation System that Preserves Speaker Identity in Academic Lectures

Related

• VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
• Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization
• Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning
• Large-scale multilingual audio visual dubbing
• DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing