Neural Language Model Pruning for Automatic Speech Recognition

2023/10/05 by Leonardo Emili, Thiago Fraga-Silva, Emili, Leonardo +7
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Music and Audio Processing #Speech Recognition and Synthesis #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2310.03424

openalex publication_date 2023/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study model pruning methods applied to Transformer-based neural network language models for automatic speech recognition. We explore three aspects of the pruning frame work, namely criterion, method and scheduler, analyzing their contribution in terms of accuracy and inference speed. To the best of our knowledge, such in-depth analyses on large-scale recognition systems has not been reported in the literature. In addition, we propose a variant of low-rank approximation suitable for incrementally compressing models, and delivering multiple models with varied target sizes. Among other results, we show that a) data-driven pruning outperforms magnitude-driven in several scenarios; b) incremental pruning achieves higher accuracy compared to one-shot pruning, especially when targeting smaller sizes; and c) low-rank approximation presents the best trade-off between size reduction and inference speed-up for moderate compression.

Related

• Accurate and Structured Pruning for Efficient Automatic Speech Recognition
• A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
• Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models
• Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition