DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model

2025/05/31 by Xueyuan Chen, Dongchao Yang, Chen, Xueyuan +13 · 2 citations
Computer Science · Medicine · Psychology · #Dysarthria #Embedding #Encoder #Intelligibility (philosophy) #Phonetics and Phonology Research #Speech Recognition and Synthesis #Speech processing #Speech synthesis #Voice activity detection #Voice and Speech Disorders

paper · pdf · doi:10.48550/arxiv.2506.00350

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Dysarthric speech reconstruction (DSR) aims to convert dysarthric speech into comprehensible speech while maintaining the speaker's identity. Despite significant advancements, existing methods often struggle with low speech intelligibility and poor speaker similarity. In this study, we introduce a novel diffusion-based DSR system that leverages a latent diffusion model to enhance the quality of speech reconstruction. Our model comprises: (i) a speech content encoder for phoneme embedding restoration via pre-trained self-supervised learning (SSL) speech foundation models; (ii) a speaker identity encoder for speaker-aware identity preservation by in-context learning mechanism; (iii) a diffusion-based speech generator to reconstruct the speech based on the restored phoneme embedding and preserved speaker identity. Through evaluations on the widely-used UASpeech corpus, our proposed model shows notable enhancements in speech intelligibility and speaker similarity.

Citations

• ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training
• WorldSimBench: Towards Video Generation Models as World Simulators
• CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
• SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
• Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
• Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction
• UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization
• StyleSpeech: Self-supervised Style Enhancing with VQ-VAE-based Pre-training for Expressive Audiobook Speech Synthesis
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Scalable Diffusion Models with Transformers
• High Fidelity Neural Audio Compression
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
• Score-Based Generative Modeling through Stochastic Differential Equations
• Learning Explicit Prosody Models and Deep Speaker Embeddings for Atypical Voice Conversion
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
• Generalized End-to-End Loss for Speaker Verification
• FiLM: Visual Reasoning with a General Conditioning Layer
• WaveNet: A Generative Model for Raw Audio

Cited by

• ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction
• Analyzing Speech Condition Effects in Dysarthric ASR: A Layer-wise Probing Study

Related

• Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation
• CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
• Enhancement of Dysarthric Speech Reconstruction by Contrastive Learning
• Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis
• Can we reconstruct a dysarthric voice with the large speech model Parler TTS?