Block Pruning For Faster Transformers

2021/09/10 by François Lagunas, Lagunas, François, Ella Charlaix +5 · 16 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.04838

EMNLP 2021. Code, hyper-parameters, evaluation results and checkpoints available at https://github.com/huggingface/nn_pruning

arxiv created 2021/09/10 · arxiv updated 2021/09/13

Abstract

Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast models. Our approach extends structured methods by considering blocks of any size and integrates this structure into the movement pruning paradigm for fine-tuning. We find that this approach learns to prune out full components of the underlying model, such as attention heads. Experiments consider classification and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models in speed and pruned models in size.

Cited by

• LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
• MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity
• CatBack: Universal Backdoor Attacks on Tabular Data via Categorical Encoding
• TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination
• VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuning
• Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
• V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Models
• High-Layer Attention Pruning with Rescaling
• Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
• Dynamic Acoustic Model Architecture Optimization in Training for ASR
• Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection
• Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
• Quantum Knowledge Distillation for Large Language Models
• Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptation
• AdaVid: Adaptive Video-Language Pretraining

Related

• A Fast Post-Training Pruning Framework for Transformers
• Improved Methods for Model Pruning and Knowledge Distillation
• MultiPruner: Balanced Structure Removal in Foundation Models
• Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference
• Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models