iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform

2022/03/04 by Takuhiro Kaneko, Kou Tanaka, Kaneko, Takuhiro +5 · 12 citations
Computer Science · #Advanced Data Compression Techniques #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2203.02395

openalex publication_date 2022/03/04 · openalex created_date 2022/08/27 · openalex updated_date 2026/07/28

Abstract

In recent text-to-speech synthesis and voice conversion systems, a mel-spectrogram is commonly applied as an intermediate representation, and the necessity for a mel-spectrogram vocoder is increasing. A mel-spectrogram vocoder must solve three inverse problems: recovery of the original-scale magnitude spectrogram, phase reconstruction, and frequency-to-time conversion. A typical convolutional mel-spectrogram vocoder solves these problems jointly and implicitly using a convolutional neural network, including temporal upsampling layers, when directly calculating a raw waveform. Such an approach allows skipping redundant processes during waveform synthesis (e.g., the direct reconstruction of high-dimensional original-scale spectrograms). By contrast, the approach solves all problems in a black box and cannot effectively employ the time-frequency structures existing in a mel-spectrogram. We thus propose iSTFTNet, which replaces some output-side layers of the mel-spectrogram vocoder with the inverse short-time Fourier transform (iSTFT) after sufficiently reducing the frequency dimension using upsampling layers, reducing the computational cost from black-box modeling and avoiding redundant estimations of high-dimensional spectrograms. During our experiments, we applied our ideas to three HiFi-GAN variants and made the models faster and more lightweight with a reasonable speech quality. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet/.

Cited by

• A Universal Harmonic Discriminator for High-quality GAN-based Vocoder
• BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspective
• E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis
• Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations
• Vocoder-Projected Feature Discriminator
• Rethinking the joint estimation of magnitude and phase for time-frequency domain neural vocoders
• A Distilled Low-Latency Neural Vocoder with Explicit Amplitude and Phase Prediction
• Spiking Vocos: An Energy-Efficient Neural Vocoder
• Is GAN Necessary for Mel-Spectrogram-based Neural Vocoder?
• Next Tokens Denoising for Speech Synthesis
• Learning Neural Vocoder from Range-Null Space Decomposition
• EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesis

Related

• ESTVocoder: An Excitation-Spectral-Transformed Neural Vocoder Conditioned on Mel Spectrogram
• HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transform
• iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN
• A Neural Denoising Vocoder for Clean Waveform Generation from Noisy Mel-Spectrogram based on Amplitude and Phase Predictions
• APNet2: High-quality and High-efficiency Neural Vocoder with Direct Prediction of Amplitude and Phase Spectra