Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning

2020/02/12 by Sejun Park, Jaeho Lee, Park, Sejun +5 · 17 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Bandwidth (computing) #Computer science #Distortion (music) #FOS: Computer and information sciences #Layer (electronics) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Magnitude (astronomy) #Neural Networks and Applications #Neural Networks and Reservoir Computing #Physics #Pruning #Residual neural network #Simple (philosophy) #Simplicity #Telecommunications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.04809

published in arXiv (Cornell University) (Cornell University) · ICLR 2020, camera ready

arxiv created 2020/02/12 · openalex publication_date 2020/02/12 · arxiv updated 2020/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants demonstrated remarkable performances for pruning modern architectures. Based on the observation that magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental results demonstrate that the proposed method consistently outperforms magnitude-based pruning on various networks, including VGG and ResNet, particularly in the high-sparsity regime. See https://github.com/alinlab/lookahead_pruning for codes.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Some NP-complete problems in quadratic and nonlinear programming
• Gradient-based learning applied to document recognition
• Bayesian Compression for Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Rethinking the Value of Network Pruning
• The State of Sparsity in Deep Neural Networks
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Learning Multiple Layers of Features from Tiny Images

Cited by

• EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
• Pruning and Quantization Impact on Graph Neural Networks
• FastCaps: A Design Methodology for Accelerating Capsule Network on Field Programmable Gate Arrays
• Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
• Sparsified State-Space Models are Efficient Highway Networks
• ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation
• Efficient Privacy-Preserving Cross-Silo Federated Learning with Multi-Key Homomorphic Encryption
• CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devices

Related

• Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Network Automatic Pruning: Start NAP and Take a Nap
• Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations
• What is the State of Neural Network Pruning?