Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data

2024/07/15 by Liang-Hsuan Tseng, Zih-Ching Chen, Tseng, Liang-Hsuan +9 · 4 citations
Engineering · #Fault Detection and Control Systems #Ferroelectric and Negative Capacitance Devices

paper · pdf · doi:10.48550/arxiv.2407.10603

Abstract

Recent advances in automatic speech recognition (ASR) often rely on large speech foundation models for generating high-quality transcriptions. However, these models can be impractical due to limited computing resources. The situation is even more severe in terms of more realistic or difficult scenarios, such as code-switching ASR (CS-ASR). To address this, we present a framework for developing more efficient models for CS-ASR through knowledge distillation using realistic speech-only data. Our proposed method, Leave No Knowledge Behind During Knowledge Distillation (K²D), leverages both the teacher model's knowledge and additional insights from a small auxiliary model. We evaluate our approach on two in-domain and two out-domain datasets, demonstrating that K²D is effective. By conducting K²D on the unlabeled realistic data, we have successfully obtained a 2-time smaller model with 5-time faster generation speed while outperforming the baseline methods and the teacher model on all the testing sets. We have made our model publicly available on Hugging Face (https://huggingface.co/andybi7676/k2d-whisper.zh-en).

Cited by

• A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
• Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review
• AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR
• ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality

Related

• Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
• Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer
• Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
• Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition
• Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models