Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning

2024/05/23 by Arnav Goel, Goel, Arnav, Medha Hira +3
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2406.00022

openalex publication_date 2024/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The field of prosody transfer in speech synthesis systems is rapidly advancing. This research is focused on evaluating learning methods for adapting pre-trained monolingual text-to-speech (TTS) models to multilingual conditions, i.e., Supervised Fine-Tuning (SFT) and Transfer Learning (TL). This comparison utilizes three distinct metrics: Mean Opinion Score (MOS), Recognition Accuracy (RA), and Mel Cepstral Distortion (MCD). Results demonstrate that, in comparison to SFT, TL leads to significantly enhanced performance, with an average MOS higher by 1.53 points, a 37.5% increase in RA, and approximately a 7.8-point improvement in MCD. These findings are instrumental in helping build TTS models for low-resource languages.

Related

• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning
• A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection