SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models

2024/06/04 by Dongchao Yang, Yang, Dongchao, Dingdong Wang +9 · 13 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Natural Language Processing Techniques #Sound (cs.SD) #Speech Recognition and Synthesis #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2406.02328

openalex publication_date 2024/06/04 · openalex created_date 2024/06/08 · openalex updated_date 2026/07/28

Abstract

In this study, we propose a simple and efficient Non-Autoregressive (NAR) text-to-speech (TTS) system based on diffusion, named SimpleSpeech. Its simpleness shows in three aspects: (1) It can be trained on the speech-only dataset, without any alignment information; (2) It directly takes plain text as input and generates speech through an NAR way; (3) It tries to model speech in a finite and compact latent space, which alleviates the modeling difficulty of diffusion. More specifically, we propose a novel speech codec model (SQ-Codec) with scalar quantization, SQ-Codec effectively maps the complex speech signal into a finite and compact latent space, named scalar latent space. Benefits from SQ-Codec, we apply a novel transformer diffusion model in the scalar latent space of SQ-Codec. We train SimpleSpeech on 4k hours of a speech-only dataset, it shows natural prosody and voice cloning ability. Compared with previous large-scale TTS models, it presents significant speech quality and generation speed improvement. Demos are released.

Cited by

• VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing
• OLaPh: Optimal Language Phonemizer
• Long-Context Speech Synthesis with Context-Aware Memory
• Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech
• Adaptive Duration Model for Text Speech Alignment
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
• DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model
• MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder
• Voice Cloning: Comprehensive Survey
• AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
• ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling

Related

• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
• Sample-Efficient Diffusion for Text-To-Speech Synthesis
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
• LatentSpeech: Latent Diffusion for Text-To-Speech Generation