Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets

2025/08/21 by Chenlin Liu, Liu, Chenlin, Ming Fang +9 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Multimodal Machine Learning Applications #Sound (cs.SD) #Topic Modeling #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2508.15442

openalex publication_date 2025/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Language Model (LM)-based Text-to-Speech (TTS) systems often generate hallucinated speech that deviates from input text. Existing mitigation strategies either demand excessive training resources or introduce significant inference latency. In this paper, we propose GFlOwNet-guided distribution AlignmenT (GOAT) for LM-based TTS, a post-training framework that mitigates hallucinations without relying on massive resources or inference cost. Specifically, we first conduct an uncertainty analysis, revealing a strong positive correlation between hallucination and model uncertainty. Based on this, we reformulate TTS generation as a trajectory flow optimization problem and introduce an enhanced Subtrajectory Balance objective together with a sharpened internal reward as target distribution. We further integrate reward temperature decay and learning rate optimization for stability and performance balance. Extensive experiments show that GOAT reduce over 50% character error rates on challenging test cases and lowering uncertainty by up to 58%, demonstrating its strong generalization ability and effectiveness.

Citations

• Qwen3 Technical Report
• Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets
• A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice
• The Llama 3 Herd of Models
• Autoregressive Speech Synthesis without Vector Quantization
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
• WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• Improving GFlowNets for Text-to-Image Diffusion Alignment
• RGFN: Synthesizable Molecular Generation Using GFlowNets
• Amortizing intractable inference in diffusion models for vision, language, and control
• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
• Ant Colony Sampling with GFlowNets for Combinatorial Optimization
• BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data
• Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
• Generative Flow Networks as Entropy-Regularized RL
• Compositional Sculpting of Iterative Generative Processes
• RepCodec: A Speech Representation Codec for Speech Tokenization
• Generalized Universal Domain Adaptation with Generative Flow Networks
• GPT-4 Technical Report
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
• CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
• Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
• Better Training of GFlowNets with Local Credit and Incomplete Trajectories
• Robust Speech Recognition via Large-Scale Weak Supervision
• A Variational Perspective on Generative Flow Networks
• Learning GFlowNets from partial episodes for improved convergence and stability
• AudioLM: a Language Modeling Approach to Audio Generation
• Generative Flow Networks for Discrete Probabilistic Modeling
• Trajectory balance: Improved credit assignment in GFlowNets
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
• Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation
• Language Models are Few-Shot Learners
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• Concrete Problems in AI Safety
• KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction
• DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics

Cited by

• Knowledge-Decoupled Functionally Invariant Path with Synthetic Personal Data for Personalized ASR

Related

• Eliminating stability hallucinations in llm-based tts models via attention guidance
• Teaming LLMs to Detect and Mitigate Hallucinations
• Pre-trained Language Models Return Distinguishable Probability Distributions to Unfaithfully Hallucinated Texts
• Alleviating Hallucinations of Large Language Models through Induced Hallucinations
• MIH-TCCT: Mitigating Inconsistent Hallucinations in LLMs via Event-Driven Text-Code Cyclic Training