Fitting New Speakers Based on a Short Untranscribed Sample

2018/02/20 by Eliya Nachmani, Nachmani, Eliya, Adam Polyak +5 · 3 citations
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1802.06984

openalex publication_date 2018/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning-based Text To Speech systems have the potential to generalize from one speaker to the next and thus require a relatively short sample of any new voice. However, this promise is currently largely unrealized. We present a method that is designed to capture a new speaker from a short untranscribed audio sample. This is done by employing an additional network that given an audio sample, places the speaker in the embedding space. This network is trained as part of the speech synthesis system using various consistency losses. Our results demonstrate a greatly improved performance on both the dataset speakers, and, more importantly, when fitting new voices, even from very short samples.

Citations

• Deep Voice: Real-time Neural Text-to-Speech
• Tacotron: Towards End-to-End Speech Synthesis

Cited by

• Sample Efficient Adaptive Text-to-Speech
• Multimodal speech synthesis architecture for unsupervised speaker adaptation
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis

Related

• Sample Efficient Adaptive Text-to-Speech
• Adapting TTS models For New Speakers using Transfer Learning
• Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with\n SampleRNN
• VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop
• GC-TTS: Few-shot Speaker Adaptation with Geometric Constraints