FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching

2025/02/16 by Wang, Hui, Liu, Shujie, Meng, Lingwei +9 · 9 citations
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2502.11128

Abstract

To advance continuous-valued token modeling and temporal-coherence enforcement, we propose FELLE, an autoregressive model that integrates language modeling with token-wise flow matching. By leveraging the autoregressive nature of language models and the generative efficacy of flow matching, FELLE effectively predicts continuous-valued tokens (mel-spectrograms). For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step, improving coherence and stability. Furthermore, to enhance synthesis quality, FELLE introduces a coarse-to-fine flow-matching mechanism, generating continuous-valued tokens hierarchically, conditioned on the language model's output. Experimental results demonstrate the potential of incorporating flow-matching techniques in autoregressive mel-spectrogram modeling, leading to significant improvements in TTS generation quality, as shown in https://aka.ms/felle.

Cited by

• Bayesian Speech synthesizers Can Learn from Multiple Teachers
• AudioEval: Automatic Dual-Perspective and Multi-Dimensional Evaluation of Text-to-Audio-Generation
• Position: Towards Responsible Evaluation for Text-to-Speech
• VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning
• Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration
• TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models
• CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
• Next Tokens Denoising for Speech Synthesis
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling

Related

• Autoregressive Speech Synthesis without Vector Quantization
• BridgeCode: A Dual Speech Representation Paradigm for Autoregressive Zero-Shot Text-to-Speech Synthesis
• Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows
• UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
• Speech Synthesis From Continuous Features Using Per-Token Latent Diffusion