Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling

2025/05/26 by Haiyang Sun, Shujie Hu, Sun, Haiyang +19 · 4 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.19669

openalex publication_date 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Zero-shot streaming text-to-speech is an important research topic in human-computer interaction. Existing methods primarily use a lookahead mechanism, relying on future text to achieve natural streaming speech synthesis, which introduces high processing latency. To address this issue, we propose SMLLE, a streaming framework for generating high-quality speech frame-by-frame. SMLLE employs a Transducer to convert text into semantic tokens in real time while simultaneously obtaining duration alignment information. The combined outputs are then fed into a fully autoregressive (AR) streaming model to reconstruct mel-spectrograms. To further stabilize the generation process, we design a Delete < Bos > Mechanism that allows the AR model to access future text introducing as minimal delay as possible. Experimental results suggest that the SMLLE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems. Samples are available on shy-98.github.io/SMLLE_demo_page/.

Citations

• Zero-Shot Text-to-Speech from Continuous Text Streams
• Moshi: a speech-text foundation model for real-time dialogue
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
• The Llama 3 Herd of Models
• Autoregressive Speech Synthesis without Vector Quantization
• High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis
• CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech
• WavLLM: Towards Robust and Adaptive Speech Large Language Model
• ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
• Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction
• SALMONN: Towards Generic Hearing Abilities for Large Language Models
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
• GPT-4 Technical Report
• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Learning Transferable Visual Models From Natural Language Supervision
• Zero-Shot Text-to-Image Generation
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech

Cited by

• Bayesian Speech synthesizers Can Learn from Multiple Teachers
• Position: Towards Responsible Evaluation for Text-to-Speech
• Next Tokens Denoising for Speech Synthesis
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling
• Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesis

Related

• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling
• Zero-Shot Text-to-Speech from Continuous Text Streams
• CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
• Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis
• VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency