Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder

2022/12/16 by Yasuda, Yusuke, Toda, Tomoki · 2 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (stat.ML) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2212.08329

Abstract

Text-to-speech synthesis (TTS) is a task to convert texts into speech. Two of the factors that have been driving TTS are the advancements of probabilistic models and latent representation learning. We propose a TTS method based on latent variable conversion using a diffusion probabilistic model and the variational autoencoder (VAE). In our TTS method, we use a waveform model based on VAE, a diffusion model that predicts the distribution of latent variables in the waveform model from texts, and an alignment model that learns alignments between the text and speech latent sequences. Our method integrates diffusion with VAE by modeling both mean and variance parameters with diffusion, where the target distribution is determined by approximation from VAE. This latent variable conversion framework potentially enables us to flexibly incorporate various latent feature extractors. Our experiments show that our method is robust to linguistic labels with poor orthography and alignment errors.

Cited by

• MultiGen: Child-Friendly Multilingual Speech Generator with LLMs
• Prompt-Unseen-Emotion: Zero-shot Expressive Speech Synthesis with Prompt-LLM Contextual Knowledge for Mixed Emotions

Related

• LatentSpeech: Latent Diffusion for Text-To-Speech Generation
• DiffVoice: Text-to-Speech with Latent Diffusion
• VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis
• SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models
• Sample-Efficient Diffusion for Text-To-Speech Synthesis