A Simplified Fully Quantized Transformer for End-to-end Speech Recognition

2019/11/09 by Alex Bie, Bharat Venkitesh, Bie, Alex +7
Computer Science · #Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #Music and Audio Processing #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.1911.03604

openalex publication_date 2019/11/09 · openalex created_date 2020/03/06 · openalex updated_date 2026/07/28

Abstract

While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use of very large neural networks, unfit for embedded use on edge devices. That being said, in this paper, we work on simplifying and compressing Transformer-based encoder-decoder architectures for the end-to-end ASR task. We empirically introduce a more compact Speech-Transformer by investigating the impact of discarding particular modules on the performance of the model. Moreover, we evaluate reducing the numerical precision of our network's weights and activations while maintaining the performance of the full-precision model. Our experiments show that we can reduce the number of parameters of the full-precision model and then further compress the model 4x by fully quantizing to 8-bit fixed point precision.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Sequence Transduction with Recurrent Neural Networks
• Sequence to Sequence Learning with Neural Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Model compression via distillation and quantization
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Efficient 8-Bit Quantization of Transformer Neural Machine Language\n Translation Model

Related

• Compute Cost Amortized Transformer for Streaming ASR
• Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank\n Transformer
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• Transformer-based Online Speech Recognition with Decoder-end Adaptive Computation Steps
• Very Deep Self-Attention Networks for End-to-End Speech Recognition