Score-Based Training for Energy-Based TTS Models

2025/05/19 by Wanli Sun, Sun, Wanli, Anton Ragni +1
Decision Sciences · Energy · #Audio and Speech Processing (eess.AS) #Construction Project Management and Performance #Energy Efficiency and Management #FOS: Computer and information sciences #FOS: Electrical engineering #Forecasting Techniques and Applications #Machine Learning (cs.LG) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.13771

openalex publication_date 2025/05/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.

Citations

• The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
• SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics
• Energy-Based Models For Speech Synthesis
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• Flow Matching for Generative Modeling
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• On Out-of-distribution Detection with Energy-based Models
• Residual Energy-Based Models for End-to-End Speech Recognition
• How to Train Your Energy-Based Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• Imitation with Neural Density Models
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Denoising Diffusion Probabilistic Models
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Generative Modeling by Estimating Gradients of the Data Distribution
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Estimating the Hessian by Back-propagating Curvature
• A Connection Between Score Matching and Denoising Autoencoders

Related

• Self-Adapting Noise-Contrastive Estimation for Energy-Based Models
• Learning Unnormalized Statistical Models via Compositional Optimization
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Conditional Noise-Contrastive Estimation of Unnormalised Models
• Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching