Lightweight End-to-end Text-to-speech Synthesis for low resource on-device applications

2025/05/12 by Vecino, Biel Tura, Gabryś, Adam, Mątwicki, Daniel +4 · 1 citation
#Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2505.07701

Abstract

Recent works have shown that modelling raw waveform directly from text in an end-to-end (E2E) fashion produces more natural-sounding speech than traditional neural text-to-speech (TTS) systems based on a cascade or two-stage approach. However, current E2E state-of-the-art models are computationally complex and memory-consuming, making them unsuitable for real-time offline on-device applications in low-resource scenarios. To address this issue, we propose a Lightweight E2E-TTS (LE2E) model that generates high-quality speech requiring minimal computational resources. We evaluate the proposed model on the LJSpeech dataset and show that it achieves state-of-the-art performance while being up to 90% smaller in terms of model parameters and 10× faster in real-time-factor. Furthermore, we demonstrate that the proposed E2E training paradigm achieves better quality compared to an equivalent architecture trained in a two-stage approach. Our results suggest that LE2E is a promising approach for developing real-time, high quality, low-resource TTS applications for on-device applications.

Citations

• Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform
• Low-data? No problem: low-resource, language-agnostic conversational text-to-speech via F0-conditioned data augmentation
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training
• NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality
• JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• ESPnet2-TTS: Extending the Edge of TTS Research
• On-device neural speech synthesis
• A Survey on Neural Speech Synthesis
• Conditional Variational Autoencoder with Adversarial Learning for\n End-to-End Text-to-Speech
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
• Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• SpeedySpeech: Efficient Neural Speech Synthesis
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• A Cyclical Post-filtering Approach to Mismatch Refinement of Neural Vocoder for Text-to-speech Systems
• Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
• Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• DurIAN: Duration Informed Attention Network For Multimodal Synthesis
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Adam: A Method for Stochastic Optimization

Cited by

• Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters

Related

• EfficientSpeech: An On-Device Text to Speech Model
• Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform
• On-device neural speech synthesis
• FLY-TTS: Fast, Lightweight and High-Quality End-to-End Text-to-Speech Synthesis
• Improving End-to-End Speech Processing by Efficient Text Data Utilization with Latent Synthesis