Can we reconstruct a dysarthric voice with the large speech model Parler TTS?

2025/06/04 by Ariadna Sanchez, Simon King, Sanchez, Ariadna +1 · 1 voice · 1 citation
Computer Science · Engineering · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Sound (cs.SD) #cs.CL #cs.SD #eess.AS #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.04397

Abstract

Speech disorders can make communication hard or even impossible for those who develop them. Personalised Text-to-Speech is an attractive option as a communication aid. We attempt voice reconstruction using a large speech model, with which we generate an approximation of a dysarthric speaker's voice prior to the onset of their condition. In particular, we investigate whether a state-of-the-art large speech model, Parler TTS, can generate intelligible speech while maintaining speaker identity. We curate a dataset and annotate it with relevant speaker and intelligibility information, and use this to fine-tune the model. Our results show that the model can indeed learn to generate from the distribution of this challenging data, but struggles to control intelligibility and to maintain consistent speaker identity. We propose future directions to improve controllability of this class of model, for the voice reconstruction task.

Citations

• Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP
• Natural language guidance of high-fidelity text-to-speech with synthetic annotations
• Creating Personalized Synthetic Voices from Articulation Impaired Speech Using Augmented Reconstruction Loss
• Speaker-Independent Acoustic-to-Articulatory Speech Inversion
• Robust Speech Recognition via Large-Scale Weak Supervision
• On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches
• UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
• Unsupervised Cross-lingual Representation Learning for Speech\n Recognition
• Improving Dysarthric Speech Intelligibility Using Cycle-consistent Adversarial Training

Cited by

• Finding My Voice: Generative Reconstruction of Disordered Speech for Automated Clinical Evaluation

Discussions

• Very excited to share that our paper has been accepted to @interspeech.bsky.social 2025! 📢 Title: Can we reconstruct a dysarthric voice with the large speech model Parler TTS? 🔗 arXiv link: www.arxi [bsky, 4 points, 1 comments]

Related

• Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation
• DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model
• CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction
• A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion
• Towards Identity Preserving Normal to Dysarthric Voice Conversion