Structured Sparsity and Weight-adaptive Pruning for Memory and Compute efficient Whisper models

2025/10/14 by Prasenjit K Mudi, Mudi, Prasenjit K, Sachan, Anshi +4
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.12666

openalex publication_date 2025/10/14 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Whisper models have achieved remarkable progress in speech recognition; yet their large size remains a bottleneck for deployment on resource-constrained edge devices. This paper proposes a framework to design fine-tuned variants of Whisper which address the above problem. Structured sparsity is enforced via the Sparse Group LASSO penalty as a loss regularizer, to reduce the number of FLOating Point operations (FLOPs). Further, a weight statistics aware pruning algorithm is proposed. We also design our custom text normalizer for WER evaluation. On Common Voice 11.0 Hindi dataset, we obtain, without degrading WER, (a) 35.4% reduction in model parameters, 14.25% lower memory consumption and 18.5% fewer FLOPs on Whisper-small, and (b) 31% reduction in model parameters, 15.29% lower memory consumption and 16.95% fewer FLOPs on Whisper-medium; and, (c) substantially outperform the state-of-the-art Iterative Magnitude Pruning based method by pruning 18.7% more parameters along with a 12.31 reduction in WER.

Citations

• Whispering in Amharic: Fine-tuning Whisper for Low-resource Language
• Fine-tuning Whisper on Low-Resource Languages for Real-World Applications
• Whisper Finetuning on Nepali Language
• Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR
• Robust Speech Recognition via Large-Scale Weak Supervision
• Towards Building Text-To-Speech Systems for the Next Billion Users
• Training Compute-Optimal Large Language Models
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Learning the Number of Neurons in Deep Networks
• Learning both Weights and Connections for Efficient Neural Networks
• A Sparse-Group Lasso

Related

• Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization
• USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models
• Sparsification via Compressed Sensing for Automatic Speech Recognition
• Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models
• Whisper-SV: Adapting Whisper for Low-data-resource Speaker Verification