An investigation into the adaptability of a diffusion-based TTS model

2023/03/03 by Chen, Haolin, Garner, Philip N.
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2303.01849

Abstract

Given the recent success of diffusion in producing natural-sounding synthetic speech, we investigate how diffusion can be used in speaker adaptive TTS. Taking cues from more traditional adaptation approaches, we show that adaptation can be included in a diffusion pipeline using conditional layer normalization with a step embedding. However, we show experimentally that, whilst the approach has merit, such adaptation alone cannot approach the performance of Transformer-based techniques. In a second experiment, we show that diffusion can be optimally combined with Transformer, with the latter taking the bulk of the adaptation load and the former contributing to improved naturalness.

Related

• Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors
• Sample-Efficient Diffusion for Text-To-Speech Synthesis
• VoiceTailor: Lightweight Plug-In Adapter for Diffusion-Based Personalized Text-to-Speech
• DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability