Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models

2023/08/07 by Park, Seungcheol, Choi, Hojun, Kang, U · 2 citations
#68T50 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7

paper · doi:10.48550/arxiv.2308.03449

Abstract

Given a pretrained encoder-based language model, how can we accurately compress it without retraining? Retraining-free structured pruning algorithms are crucial in pretrained language model compression due to their significantly reduced pruning cost and capability to prune large language models. However, existing retraining-free algorithms encounter severe accuracy degradation, as they fail to handle pruning errors, especially at high compression rates. In this paper, we propose K-prune (Knowledge-preserving pruning), an accurate retraining-free structured pruning algorithm for pretrained encoder-based language models. K-prune focuses on preserving the useful knowledge of the pretrained model to minimize pruning errors through a carefully designed iterative pruning process composed of knowledge measurement, knowledge-preserving mask search, and knowledge-preserving weight-tuning. As a result, K-prune shows significant accuracy improvements up to 58.02%p higher F1 score compared to existing retraining-free pruning algorithms under a high compression rate of 80% on the SQuAD benchmark without any retraining process.

Cited by

• Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
• Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Information

Related

• Pruning Pre-trained Language Models Without Fine-Tuning
• Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models
• Pruning Pre-trained Language Models with Principled Importance and Self-regularization
• IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
• Reconstruct the Pruned Model without Any Retraining