Efficient Speech Representation Learning with Low-Bit Quantization

2022/12/14 by Yeh, Ching-Feng, Hsu, Wei-Ning, Tomasello, Paden +1
#Audio and Speech Processing (eess.AS) #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Electrical engineering #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2301.00652

Abstract

With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity. In effort to improve the efficiency for these models, we apply and investigate recent quantization techniques on speech representation learning models. The quantization techniques were evaluated on the SUPERB benchmark. On the ASR task, with aggressive quantization to 1 bit, we achieved 86.32% storage reduction (184.42 -> 25.23), 88% estimated runtime reduction (1.00 -> 0.12) with increased word error rate (7.06 -> 15.96). In comparison with DistillHuBERT which also aims for model compression, the 2-bit configuration yielded slightly smaller storage (35.84 vs. 46.98), better word error rate (12.68 vs. 13.37) and more efficient estimated runtime (0.15 vs. 0.73).

Related

• 4-bit Conformer with Native Quantization Aware Training for Speech Recognition
• Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precision
• Sub-8-bit quantization for on-device speech recognition: a regularization-free approach
• On the efficient representation and execution of deep acoustic models
• Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition Models