Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

2022/06/05 by Levkovitch, Alon, Nachmani, Eliya, Wolf, Lior
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2206.02246

Abstract

We present a novel way of conditioning a pretrained denoising diffusion speech model to produce speech in the voice of a novel person unseen during training. The method requires a short (~3 seconds) sample from the target person, and generation is steered at inference time, without any training steps. At the heart of the method lies a sampling process that combines the estimation of the denoising model with a low-pass version of the new speaker's sample. The objective and subjective evaluations show that our sampling method can generate a voice similar to that of the target speaker in terms of frequency, with an accuracy comparable to state-of-the-art methods, and without training.

Related

• Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation
• Zero-shot Voice Conversion with Diffusion Transformers
• E1 TTS: Simple and Fast Non-Autoregressive TTS
• Discrete-Time Diffusion-Like Models for Speech Synthesis
• Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion