SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model

2021/04/02 by Edresson Casanova, Casanova, Edresson, Christopher Shulby +15 · 12 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2104.05557

Accepted on Interspeech 2021

arxiv created 2021/06/15 · arxiv updated 2021/06/17

Abstract

In this paper, we propose SC-GlowTTS: an efficient zero-shot multi-speaker text-to-speech model that improves similarity for speakers unseen during training. We propose a speaker-conditional architecture that explores a flow-based decoder that works in a zero-shot scenario. As text encoders, we explore a dilated residual convolutional-based encoder, gated convolutional-based encoder, and transformer-based encoder. Additionally, we have shown that adjusting a GAN-based vocoder for the spectrograms predicted by the TTS model on the training dataset can significantly improve the similarity and speech quality for new speakers. Our model converges using only 11 speakers, reaching state-of-the-art results for similarity with new speakers, as well as high speech quality.

Cited by

• Adaptive Edge-Cloud Inference for Speech-to-Action Systems Using ASR and Large Language Models
• Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters
• RephraseTTS: Dynamic Length Text based Speech Insertion with Speaker Style Transfer
• Lessons Learnt: Revisit Key Training Strategies for Effective Speech Emotion Recognition in the Wild
• DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
• Enhancing Target Speaker Extraction with Explicit Speaker Consistency Modeling
• ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition
• Speaker-agnostic Emotion Vector for Cross-speaker Emotion Intensity Control
• Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulations
• STOPA: A Database of Systematic VariaTion Of DeePfake Audio for Open-Set Source Tracing and Attribution
• Improving Noise Robustness of LLM-based Zero-shot TTS via Discrete Acoustic Token Denoising
• Voice Cloning: Comprehensive Survey

Related

• Glow-WaveGAN 2: High-quality Zero-shot Text-to-speech Synthesis and Any-to-any Voice Conversion
• Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios
• Multi-modal Adversarial Training for Zero-Shot Voice Cloning
• Lightweight Zero-shot Text-to-Speech with Mixture of Adapters