Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation

2021/03/17 by Md Akmal Haidar, Chao Xing, Haidar, Md Akmal +3
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Natural Language Processing Techniques #Speech Recognition and Synthesis #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2103.09903

arxiv created 2021/03/17 · openalex publication_date 2021/03/17 · arxiv updated 2021/03/19 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

End-to-end automatic speech recognition (ASR), unlike conventional ASR, does not have modules to learn the semantic representation from speech encoder. Moreover, the higher frame-rate of speech representation prevents the model to learn the semantic representation properly. Therefore, the models that are constructed by the lower frame-rate of speech encoder lead to better performance. For Transformer-based ASR, the lower frame-rate is not only important for learning better semantic representation but also for reducing the computational complexity due to the self-attention mechanism which has O(n²) order of complexity in both training and inference. In this paper, we propose a Transformer-based ASR model with the time reduction layer, in which we incorporate time reduction layer inside transformer encoder layers in addition to traditional sub-sampling methods to input features that further reduce the frame-rate. This can help in reducing the computational cost of the self-attention process for training and inference with performance improvement. Moreover, we introduce a fine-tuning approach for pre-trained ASR models using self-knowledge distillation (S-KD) which further improves the performance of our ASR model. Experiments on LibriSpeech datasets show that our proposed methods outperform all other Transformer-based ASR systems. Furthermore, with language model (LM) fusion, we achieve new state-of-the-art word error rate (WER) results for Transformer-based ASR models with just 30 million parameters trained without any external data.

Related

• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning
• Streaming automatic speech recognition with the transformer model
• Attention-based ASR with Lightweight and Dynamic Convolutions
• Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model
• Investigation of Speaker-adaptation methods in Transformer based ASR