2021/12/16 by Derek Tam, Surafel M. Lakew, Tam, Derek +7 · 2 citations
Arts and Humanities · Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Music and Audio Processing #Subtitles and Audiovisual Media
paper · pdf · doi:10.48550/arxiv.2112.08548
openalex publication_date 2021/12/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structure of the source into the target language to achieve audiovisual coherence. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source speech segments. In this work, we propose implicit and explicit modeling approaches to integrate isochrony information into neural machine translation. Experiments on English-German/French language pairs with automatic metrics show that the simplest of the considered approaches works best. Results are confirmed by human evaluations of translations and dubbed videos.