Sparsity in Deep Learning: Pruning and growth for efficient inference\n and training in neural networks

2021/01/31 by Torsten Hoefler, Dan Alistarh, Hoefler, Torsten +7 · 48 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Machine Learning and ELM #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.2102.00554

openalex publication_date 2021/01/31 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

The growing energy and performance costs of deep learning have driven the\ncommunity to reduce the size of neural networks by selectively pruning\ncomponents. Similarly to their biological counterparts, sparse networks\ngeneralize just as well, if not better than, the original dense networks.\nSparsity can reduce the memory footprint of regular networks to fit mobile\ndevices, as well as shorten training time for ever growing networks. In this\npaper, we survey prior work on sparsity in deep learning and provide an\nextensive tutorial of sparsification for both inference and training. We\ndescribe approaches to remove and add elements of neural networks, different\ntraining strategies to achieve model sparsity, and mechanisms to exploit\nsparsity in practice. Our work distills ideas from more than 300 research\npapers and provides guidance to practitioners who wish to utilize sparsity\ntoday, as well as to researchers whose goal is to push the frontier forward. We\ninclude the necessary background on mathematical methods in sparsification,\ndescribe phenomena such as early structure adaptation, the intricate relations\nbetween sparsity and the training process, and show techniques for achieving\nacceleration on real hardware. We also define a metric of pruned parameter\nefficiency that could serve as a baseline for comparison of different sparse\nnetworks. We close by speculating on how sparsity can improve future workloads\nand outline major open problems in the field.\n

Cited by

• A Proximal-Gradient Method for Solving Regularized Optimization Problems with General Constraints
• gMAGNUS: Fast SpGEMM on GPUs for Irregular Matrices via Hierarchical Multisplit
• TOGGLE: Temporal Logic-Guided Large Language Model Compression for Edge
• Optimized Architectures for Kolmogorov-Arnold Networks
• SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale
• Spatial Spiking Neural Networks Enable Efficient and Robust Temporal Computation
• On-Demand Multi-Task Sparsity for Efficient Large-Model Deployment on Edge Devices
• ModHiFi: Identifying High Fidelity predictive components for Model Modification
• FastForward Pruning: Efficient LLM Pruning via Single-Step Reinforcement Learning
• Weight Variance Amplifier Improves Accuracy in High-Sparsity One-Shot Pruning
• Efficient Mathematical Reasoning Models via Dynamic Pruning and Knowledge Distillation
• Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training
• FuseFlow: A Fusion-Centric Compilation Framework for Sparse Deep Learning on Streaming Dataflow
• Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies
• maxVSTAR: Maximally Adaptive Vision-Guided CSI Sensing with Closed-Loop Edge Model Adaptation for Robust Human Activity Recognition
• The Curious Case of In-Training Compression of State Space Models
• Architecture, Simulation and Software Stack to Support Post-CMOS Accelerators: The ARCHYTAS Project
• Adaptive Training of INRs via Pruning and Densification
• Frustratingly Easy Task-aware Pruning for Large Language Models
• Pruning and Quantization Impact on Graph Neural Networks
• Mixed-Precision Quantization for Language Models: Techniques and Prospects
• WARP-LUTs -- Walsh-Assisted Relaxation for Probabilistic Look Up Tables
• Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfe
• Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
• Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
• Shift-Invariant Attribute Scoring for Kolmogorov-Arnold Networks via Shapley Value
• pyGinkgo: A Sparse Linear Algebra Operator Framework for Python
• The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
• CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
• Growing Winning Subnetworks, Not Pruning Them: A Paradigm for Density Discovery in Sparse Neural Networks
• BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression
• Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization
• Feedback promotes efficient-coding while attenuating bias in recurrent neural networks
• StructPrune: Structured Global Pruning asymptotics with O(√(N)) GPU Memory
• SnipSnap: A Joint Compression Format and Dataflow Co-Optimization Framework for Efficient Sparse LLM Accelerator Design
• Multi-resolution Enhancement for Full Spectrum Neural Representations
• Balancing Sparse RNNs with Hyperparameterization Benefiting Meta-Learning
• The Quest for Universal Master Key Filters in DS-CNNs
• Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage
• Cache Management for Mixture-of-Experts LLMs -- extended version
• AI Compute Architecture and Evolution Trends
• Saddle Hierarchy in Dense Associative Memory
• Brain-inspired wiring economics for artificial neural networks
• One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression
• Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNs
• Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
• The Ubiquitous Sparse Matrix-Matrix Products
• Learning Explainable Stock Predictions with Tweets Using Mixture of Experts

Related

• The State of Sparsity in Deep Neural Networks
• Layer Sparsity in Neural Networks
• Experiments on Properties of Hidden Structures of Sparse Neural Networks
• Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
• Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers