Effective Model Pruning
2025/09/30 by Wang, Yixuan, Guralnik, Dan, Akbari, Saiedeh +1
#FOS: Computer and information sciences #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2509.25606
Abstract
We introduce Effective Model Pruning (EMP), a context-agnostic, parameter-free rule addressing a fundamental question about pruning: how many entries to keep. EMP does not prescribe how to score the parameters or prune the models; instead, it supplies a universal adaptive threshold that can be applied to any pruning criterion: weight magnitude, attention score, KAN importance score, or even feature-level signals such as image pixel, and used on structural parts or weights of the models. Given any score vector s, EMP maps s to a built-in effective number Neff which is inspired by the Inverse Simpson index of contributors. Retaining the Neff highest scoring entries and zeroing the remainder yields sparse models with performance comparable to the original dense networks across MLPs, CNNs, Transformers/LLMs, and KAN, in our experiments. By leveraging the geometry of the simplex, we derive a tight lower bound on the preserved mass seff (the sum of retained scores) over the corresponding ordered probability simplex associated with the score vector s. We further verify the effectiveness of Neff by pruning the model with a scaled threshold \beta*Neff across a variety of criteria and models. Experiments suggest that the default \beta = 1 yields a robust threshold for model pruning while \beta not equal to 1 still serves as an optional adjustment to meet specific sparsity requirements.
Citations
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- KAN: Kolmogorov-Arnold Networks
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
- VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
- Qwen Technical Report
- A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations
- A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- A Simple and Effective Pruning Approach for Large Language Models
- LLM-Pruner: On the Structural Pruning of Large Language Models
- Visual Instruction Tuning
- LLaMA: Open and Efficient Foundation Language Models
- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
- Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Denoising Diffusion Probabilistic Models
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Are Sixteen Heads Really Better than One?
- The State of Sparsity in Deep Neural Networks
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Learning Sparse Neural Networks through L0 Regularization
- Learning Efficient Convolutional Networks through Network Slimming
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
- Attention Is All You Need
- Pointer Sentinel Mixture Models
- Pruning Filters for Efficient ConvNets
- Deep Residual Learning for Image Recognition
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- ImageNet classification with deep convolutional neural networks
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Related