Sample-Efficient Diffusion for Text-To-Speech Synthesis

2024/09/01 by Justin Lovelace, Soham Ray, Lovelace, Justin +7 · 1 citation
Computer Science · #Speech Recognition and Synthesis #Speech and Audio Processing #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2409.03717

Abstract

This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.

Cited by

• SpeechOp: Inference-Time Task Composition for Generative Speech Processing

Related

• SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
• Autoregressive Diffusion Transformer for Text-to-Speech Synthesis
• Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
• Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study