Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis

2025/06/30 by Paul Mayer, Mayer, Paul, Florian Lux +10
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Phonetics and Phonology Research #Speech Recognition and Synthesis #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.00227

openalex publication_date 2025/06/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While generative methods have progressed rapidly in recent years, generating expressive prosody for an utterance remains a challenging task in text-to-speech synthesis. This is particularly true for systems that model prosody explicitly through parameters such as pitch, energy, and duration, which is commonly done for the sake of interpretability and controllability. In this work, we investigate the effectiveness of stochastic methods for this task, including Normalizing Flows, Conditional Flow Matching, and Rectified Flows. We compare these methods to a traditional deterministic baseline, as well as to real human realizations. Our extensive subjective and objective evaluations demonstrate that stochastic methods produce natural prosody on par with human speakers by capturing the variability inherent in human speech. Further, they open up additional controllability options by allowing the sampling temperature to be tuned.

Citations

• ASVspoof 5: Design, Collection and Validation of Resources for Spoofing, Deepfake, and Adversarial Attack Detection Using Crowdsourced Speech
• ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech
• Meta Learning Text-to-Speech Synthesis in over 7000 Languages
• Should you use a probabilistic duration model in TTS? Probably! Especially for spontaneous speech
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• The IMS Toucan System for the Blizzard Challenge 2023
• Better speech synthesis through scaling
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
• A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Scalable Diffusion Models with Transformers
• Flow Matching for Generative Modeling
• Rectified Flow: A Marginal Preserving Approach to Optimal Transport
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• PoeticTTS -- Controllable Poetry Reading for Literary Studies
• DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders
• Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech
• NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality
• Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021
• ADEPT: A Dataset for Evaluating Prosody Transfer
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• SpeechBrain: A General-Purpose Speech Toolkit
• FastPitch: Parallel Text-to-speech with Pitch Prediction
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• Variational Inference with Normalizing Flows
• Multiple Comparisons Using Rank Sums
• Use of Ranks in One-Criterion Variance Analysis
• Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey
• Divergence measures based on the Shannon entropy

Related

• Stochastic Pitch Prediction Improves the Diversity and Naturalness of Speech in Glow-TTS
• Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis
• VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching
• DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions
• Prosodic Parameter Manipulation in TTS generated speech for Controlled Speech Generation