A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data

2025/06/10 by C.-Y. Chou, Chou, Cheng-Kang, Chan-Jan Hsu +10
Engineering · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Fault Detection and Control Systems #Sound (cs.SD) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2506.11130

openalex publication_date 2025/06/10 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

We propose a self-refining framework that enhances ASR performance with only unlabeled datasets. The process starts with an existing ASR model generating pseudo-labels on unannotated speech, which are then used to train a high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs are bootstrapped into the original ASR system, completing the closed-loop self-improvement cycle. We demonstrated the effectiveness of the framework on Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a moderate amount of text data, and synthetic content from the AI models, we adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching benchmarks compared to Whisper. Results highlight the framework as a compelling alternative to pseudo-labeling self-distillation approaches and provides a practical pathway for improving ASR performance in low-resource or domain-specific settings.

Citations

• Whispering in Amharic: Fine-tuning Whisper for Low-resource Language
• Improving the Inclusivity of Dutch Speech Recognition by Fine-tuning Whisper on the JASMIN-CGN Corpus
• BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights
• Fine-tuning Whisper on Low-Resource Languages for Real-World Applications
• Efficient Adaptation of Multilingual Models for Japanese ASR
• Whisper Finetuning on Nepali Language
• Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gap
• Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition
• Whispering in Norwegian: Navigating Orthographic and Dialectic Challenges
• OLMo: Accelerating the Science of Language Models
• Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision
• Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages
• Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models
• FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec
• Can Whisper perform speech-based in-context learning?
• Matcha-TTS: A fast TTS architecture with conditional flow matching
• Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning
• Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
• Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
• LLaMA: Open and Efficient Foundation Language Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• High Fidelity Neural Audio Compression
• Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems
• Common Voice: A Massively-Multilingual Speech Corpus
• Machine Speech Chain with One-shot Speaker Adaptation
• Listening while Speaking: Speech Chain by Deep Learning

Related

• Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gap
• Improving Low Resource Code-switched ASR using Augmented Code-switched\n TTS
• EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition
• A Survey on Non-Intrusive ASR Refinement: From Output-Level Correction to Full-Model Distillation
• Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLM