Statistical Parametric Speech Synthesis Using Generative Adversarial Networks Under A Multi-task Learning Framework

2017/07/06 by Shan Yang, Lei Xie, Yang, Shan +11 · 1 citation
Computer Science · #FOS: Computer and information sciences #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.1707.01670

openalex publication_date 2017/07/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we aim at improving the performance of synthesized speech in statistical parametric speech synthesis (SPSS) based on a generative adversarial network (GAN). In particular, we propose a novel architecture combining the traditional acoustic loss function and the GAN's discriminative loss under a multi-task learning (MTL) framework. The mean squared error (MSE) is usually used to estimate the parameters of deep neural networks, which only considers the numerical difference between the raw audio and the synthesized one. To mitigate this problem, we introduce the GAN as a second task to determine if the input is a natural speech with specific conditions. In this MTL framework, the MSE optimization improves the stability of GAN, and at the same time GAN produces samples with a distribution closer to natural speech. Listening tests show that the multi-task architecture can generate more natural speech that satisfies human perception than the conventional methods.

Citations

• Tacotron: Towards End-to-End Speech Synthesis
• Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Cited by

• Generative adversarial networks for speech processing: A review

Related

• Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks
• Multi-task WaveNet: A Multi-task Generative Model for Statistical Parametric Speech Synthesis without Fundamental Frequency Conditions
• High Fidelity Speech Synthesis with Adversarial Networks
• TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
• Waveform generation for text-to-speech synthesis using pitch-synchronous\n multi-scale generative adversarial networks