Exploring Weight Importance and Hessian Bias in Model Pruning

2020/06/19 by Mingchen Li, Li, Mingchen, Yahya Sattar +5 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.10903

28 pages

arxiv created 2020/06/19 · openalex publication_date 2020/06/19 · arxiv updated 2020/06/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Model pruning is an essential procedure for building compact and computationally-efficient machine learning models. A key feature of a good pruning algorithm is that it accurately quantifies the relative importance of the model weights. While model pruning has a rich history, we still don't have a full grasp of the pruning mechanics even for relatively simple problems involving linear models or shallow neural nets. In this work, we provide a principled exploration of pruning by building on a natural notion of importance. For linear models, we show that this notion of importance is captured by covariance scaling which connects to the well-known Hessian-based pruning. We then derive asymptotic formulas that allow us to precisely compare the performance of different pruning methods. For neural networks, we demonstrate that the importance can be at odds with larger magnitudes and proper initialization is critical for magnitude-based pruning. Specifically, we identify settings in which weights become more important despite becoming smaller, which in turn leads to a catastrophic failure of magnitude-based pruning. Our results also elucidate that implicit regularization in the form of Hessian structure has a catalytic role in identifying the important weights, which dictate the pruning performance.

Citations

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Reconciling modern machine learning practice and the bias-variance trade-off
• Path Length Bounds for Gradient Descent and Flow
• The generalization error of random features regression: Precise asymptotics and double descent curve
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need

Cited by

• Optimal Implicit Bias in Linear Regression

Related

• Effective Model Pruning
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• Pruning has a disparate impact on model accuracy
• "Understanding Robustness Lottery": A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches
• The Generalization-Stability Tradeoff In Neural Network Pruning