Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Loss

2020/10/22 by Jiatong Shi, Shuai Guo, Shi, Jiatong +7 · 4 citations
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #cs.LG #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2010.12024

Accepted by ICASSP2021

openalex publication_date 2020/10/22 · openalex created_date 2020/10/29 · arxiv created 2021/02/26 · arxiv updated 2021/03/01 · openalex updated_date 2026/07/28

Abstract

The neural network (NN) based singing voice synthesis (SVS) systems require sufficient data to train well and are prone to over-fitting due to data scarcity. However, we often encounter data limitation problem in building SVS systems because of high data acquisition and annotation costs. In this work, we propose a Perceptual Entropy (PE) loss derived from a psycho-acoustic hearing model to regularize the network. With a one-hour open-source singing voice database, we explore the impact of the PE loss on various mainstream sequence-to-sequence models, including the RNN-based, transformer-based, and conformer-based models. Our experiments show that the PE loss can mitigate the over-fitting problem and significantly improve the synthesized singing quality reflected in objective and subjective evaluations.

Citations

• Conformer: Convolution-augmented Transformer for Speech Recognition
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

Cited by

• Robust Training of Singing Voice Synthesis Using Prior and Posterior Uncertainty
• PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning
• CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
• SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment

Related

• A Neural Parametric Singing Synthesizer
• RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis
• NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit
• Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer
• Singing voice synthesis based on convolutional neural networks