Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters

2024/01/10 by Ken‐ichi Fujita, Hiroshi Satō, Fujita, Kenichi +11 · 3 citations
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2401.05111

openalex publication_date 2024/01/10 · openalex created_date 2024/01/13 · openalex updated_date 2026/07/28

Abstract

The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method. We incorporated adapters into the SSL model, which we fine-tuned with the TTS model using noisy reference speech. In addition, to further improve performance, we adopted a speech enhancement (SE) front-end. With these improvements, our proposed SSL-based zero-shot TTS achieved high-quality speech synthesis with noisy reference speech. Through the objective and subjective evaluations, we confirmed that the proposed method is highly robust to noise in reference speech, and effectively works in combination with SE.

Cited by

• Generic Speech Enhancement with Self-Supervised Representation Space Loss
• Improving Noise Robustness of LLM-based Zero-shot TTS via Discrete Acoustic Token Denoising
• Voice Cloning: Comprehensive Survey

Related

• An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
• Lightweight Zero-shot Text-to-Speech with Mixture of Adapters
• Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron
• Rep2wav: Noise Robust text-to-speech Using self-supervised representations
• HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis