Efficient Neural and Numerical Methods for High-Quality Online Speech Spectrogram Inversion via Gradient Theorem

2025/05/30 by Andres Fernandez, Fernandez, Andres, Juan Azcarreta +5 · 1 citation
Computer Science · #FOS: Computer and information sciences #Image and Signal Denoising Methods #Machine Learning (cs.LG) #Neural Networks and Applications #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2505.24498

openalex publication_date 2025/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work in online speech spectrogram inversion effectively combines Deep Learning with the Gradient Theorem to predict phase derivatives directly from magnitudes. Then, phases are estimated from their derivatives via least squares, resulting in a high quality reconstruction. In this work, we introduce three innovations that drastically reduce computational cost, while maintaining high quality: Firstly, we introduce a novel neural network architecture with just 8k parameters, 30 times smaller than previous state of the art. Secondly, increasing latency by 1 hop size allows us to further halve the cost of the neural inference step. Thirdly, we we observe that the least squares problem features a tridiagonal matrix and propose a linear-complexity solver for the least squares step that leverages tridiagonality and positive-semidefiniteness, achieving a speedup of several orders of magnitude. We release samples online.

Citations

• A Flexible Online Framework for Projection-Based STFT Phase Retrieval
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
• SubSpectral Normalization for Neural Audio Data Processing
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement
• On the Variance of the Adaptive Learning Rate and Beyond
• GANSynth: Adversarial Neural Audio Synthesis
• Phase reconstruction from amplitude spectrograms based on von-Mises-distribution deep neural network
• End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
• WaveNet: A Generative Model for Raw Audio
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Estimating a Signal from a Magnitude Spectrogram via Convex Optimization
• PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements\n via Convex Programming
• PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming
• The importance of phase in speech enhancement

Cited by

• Revisiting Vocos: That Phasiness Business in Time-Frequency Neural Vocoding

Related

• Spectrogram Inversion for Audio Source Separation via Consistency, Mixing, and Magnitude Constraints
• A Survey of Deep Learning for Complex Speech Spectrograms
• Deep Griffin-Lim Iteration
• Mathematical Vocoder Algorithm : Modified Spectral Inversion for Efficient Neural Speech Synthesis
• Phase reconstruction from amplitude spectrograms based on von-Mises-distribution deep neural network