Real-Time Streaming Mel Vocoding with Generative Flow Matching

2025/09/18 by Welker, Simon, Peer, Tal, Gerkmann, Timo · 1 citation
#Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2509.15085

Abstract

The task of Mel vocoding, i.e., the inversion of a Mel magnitude spectrogram to an audio waveform, is still a key component in many text-to-speech (TTS) systems today. Based on generative flow matching, our prior work on generative STFT phase retrieval (DiffPhase), and the pseudoinverse operator of the Mel filterbank, we develop MelFlow, a streaming-capable generative Mel vocoder for speech sampled at 16 kHz with an algorithmic latency of only 32 ms and a total latency of 48 ms. We show real-time streaming capability at this latency not only in theory, but in practice on a consumer laptop GPU. Furthermore, we show that our model achieves substantially better PESQ and SI-SDR values compared to well-established not streaming-capable baselines for Mel vocoding including HiFi-GAN.

Citations

• Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency
• FlowDec: A flow-based full-band general audio codec with high perceptual quality
• Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
• Open-Source Conversational AI with SpeechBrain 1.0
• FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter
• EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
• A Flexible Online Framework for Projection-Based STFT Phase Retrieval
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
• Framewise WaveGAN: High Speed Adversarial Vocoder in Time Domain with Very Low Computational Complexity
• Robust Speech Recognition via Large-Scale Weak Supervision
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain
• A Streamwise GAN Vocoder for Wideband Speech Coding at Very Low Bit Rate
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• SubSpectral Normalization for Neural Audio Data Processing
• Score-Based Generative Modeling through Stochastic Differential Equations
• StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalization
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• SDR - half-baked or well done?
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Phase retrieval by iterated projections

Cited by

• Real-Time Streamable Generative Speech Restoration with Flow Matching

Related

• Real-Time Streamable Generative Speech Restoration with Flow Matching
• UniverSR: Unified and Versatile Audio Super-Resolution via Vocoder-Free Flow Matching
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• SynthVC: Leveraging Synthetic Data for End-to-End Low Latency Streaming Voice Conversion
• WaveGlow: A Flow-based Generative Network for Speech Synthesis