DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

2026/07/31 by Junwon Moon, Yejin Lee, Seungbeom Kim +4
Engineering · Computer Science · #eess.AS #cs.CL

paper · pdf

ICML 2026 SPIGM Workshop

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3x faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation.

Citations

• Fast-dLLM v2: Efficient Block-Diffusion LLM
• VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
• FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbot
• Dream 7B: Diffusion Large Language Models
• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
• Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
• TESS 2: A Large-Scale Generalist Diffusion Language Model
• Large Language Diffusion Models
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation
• Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
• Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
• Simple and Effective Masked Diffusion Language Models
• Simplified and Generalized Masked Diffusion for Discrete Data
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
• LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• SoundStorm: Efficient Parallel Audio Generation
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Robust Speech Recognition via Large-Scale Weak Supervision
• Continuous diffusion for categorical data
• SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• Diffusion-LM Improves Controllable Text Generation
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Structured Denoising Diffusion Models in Discrete State-Spaces
• LoRA: Low-Rank Adaptation of Large Language Models
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Conformer: Convolution-augmented Transformer for Speech Recognition
• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• The Curious Case of Neural Text Degeneration
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech