Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech

2024/06/08 by Shivam Mehta, Harm Lameris, Mehta, Shivam +9 · 3 citations
Computer Science · Psychology · #68T07 #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #H.5.5 #Human-Computer Interaction (cs.HC) #I.2.6 #I.2.7 #Phonetics and Phonology Research #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2406.05401

openalex publication_date 2024/06/08 · openalex created_date 2024/06/12 · openalex updated_date 2026/07/28

Abstract

Converting input symbols to output audio in TTS requires modelling the durations of speech sounds. Leading non-autoregressive (NAR) TTS models treat duration modelling as a regression problem. The same utterance is then spoken with identical timings every time, unlike when a human speaks. Probabilistic models of duration have been proposed, but there is mixed evidence of their benefits. However, prior studies generally only consider speech read aloud, and ignore spontaneous speech, despite the latter being both a more common and a more variable mode of speaking. We compare the effect of conventional deterministic duration modelling to durations sampled from a powerful probabilistic model based on conditional flow matching (OT-CFM), in three different NAR TTS approaches: regression-based, deep generative, and end-to-end. Across four different corpora, stochastic duration modelling improves probabilistic NAR TTS approaches, especially for spontaneous speech. Please see https://shivammehta25.github.io/prob_dur/ for audio and resources.

Cited by

• Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
• FNH-TTS: Mixture-of-Experts Duration Modeling for Robust Neural Speech Synthesis
• Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis

Related

• Expressive, Variable, and Controllable Duration Modelling in TTS
• Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource\n Highly Expressive Speech
• Neural Network-Based Modeling of Phonetic Durations
• Total-Duration-Aware Duration Modeling for Text-to-Speech Systems
• FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech