Bridging the gap between training and inference in LM-based TTS models

2025/09/21 by Ruonan Zhang, L. Mu, Zhang, Ruonan +5
Computer Science · Engineering · #Advanced Data Processing Techniques #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Intelligent Tutoring Systems and Adaptive Learning #Sound (cs.SD) #Speech and dialogue systems #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2509.17021

openalex publication_date 2025/09/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in text-to-speech (TTS) have shown that language model (LM) based systems offer competitive performance compared to traditional approaches. However, in training, TTS models use ground-truth (GT) tokens as prefixes to predict the next token, while in inference these tokens are not available, a gap between training and inference that is often neglected. In this study, we propose a prompt-guided hybrid training scheme to mitigate exposure bias in popular LM-based TTS systems. Our core idea is to adopt a hybrid training paradigm that combines teacher forcing with free running, thereby introducing self-generated tokens into the training process. This makes the training mode more consistent with inference, reducing the training-inference gap. In addition, we incorporate an EOS prediction mechanism during training to detect incorrect sequence termination and adaptively control the free running process. Experimental results provide a comprehensive evaluation of the impact of exposure bias on LM-based TTS, and demonstrate that our method effectively narrows the training-inference gap, thereby improving the quality of synthesized long-form speech.

Citations

• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• Preference Alignment Improves Language Model-Based TTS
• Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization
• Autoregressive Speech Synthesis without Vector Quantization
• ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
• Understanding and Bridging the Modality Gap for Speech Translation
• StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis
• On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
• Parallel Scheduled Sampling
• Bridging the Gap between Training and Inference for Neural Machine Translation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Related

• Training Text-to-Speech Model with Purely Synthetic Data: Feasibility, Sensitivity, and Generalization Capability
• LAST: Language Model Aware Speech Tokenization
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
• A study on the efficacy of model pre-training in developing neural text-to-speech system
• Speech Model Pre-training for End-to-End Spoken Language Understanding