CoDERT: Distilling Encoder Representations with Co-learning for\n Transducer-based Speech Recognition

2021/06/14 by Rupak Vignesh Swaminathan, Swaminathan, Rupak Vignesh, Brian King +7 · 1 citation
Computer Science · #Speech Recognition and Synthesis #Speech and Audio Processing #Music and Audio Processing

paper · pdf · doi:10.48550/arxiv.2106.07734

Abstract

We propose a simple yet effective method to compress an RNN-Transducer\n(RNN-T) through the well-known knowledge distillation paradigm. We show that\nthe transducer's encoder outputs naturally have a high entropy and contain rich\ninformation about acoustically similar word-piece confusions. This rich\ninformation is suppressed when combined with the lower entropy decoder outputs\nto produce the joint network logits. Consequently, we introduce an auxiliary\nloss to distill the encoder logits from a teacher transducer's encoder, and\nexplore training strategies where this encoder distillation works effectively.\nWe find that tandem training of teacher and student encoders with an inplace\nencoder distillation outperforms the use of a pre-trained and static teacher\ntransducer. We also report an interesting phenomenon we refer to as implicit\ndistillation, that occurs when the teacher and student encoders share the same\ndecoder. Our experiments show 5.37-8.4% relative word error rate reductions\n(WERR) on in-house test sets, and 5.05-6.18% relative WERRs on LibriSpeech test\nsets.\n

Cited by

• Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization

Related

• Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss
• Efficient Knowledge Distillation for RNN-Transducer Models
• Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
• Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
• Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR