Training Sparse Neural Networks using Compressed Sensing

2020/08/21 by Jonathan W. Siegel, Jianhong Chen, Siegel, Jonathan W. +3
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #Electrical and Bioimpedance Tomography #FOS: Computer and information sciences #I.5.1 #Machine Learning and ELM #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2008.09661

openalex publication_date 2020/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pruning the weights of neural networks is an effective and widely-used technique for reducing model size and inference complexity. We develop and test a novel method based on compressed sensing which combines the pruning and training into a single step. Specifically, we utilize an adaptively weighted ℓ¹ penalty on the weights during training, which we combine with a generalization of the regularized dual averaging (RDA) algorithm in order to train sparse neural networks. The adaptive weighting we introduce corresponds to a novel regularizer based on the logarithm of the absolute value of the weights. We perform a series of ablation studies demonstrating the improvement provided by the adaptive weighting and generalized RDA algorithm. Furthermore, numerical experiments on the CIFAR-10, CIFAR-100, and ImageNet datasets demonstrate that our method 1) trains sparser, more accurate networks than existing state-of-the-art methods; 2) can be used to train sparse networks from scratch, i.e. from a random initialization, as opposed to initializing with a well-trained base model; 3) acts as an effective regularizer, improving generalization accuracy.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• A Sparse-Group Lasso
• Gradient-based learning applied to document recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Primal-dual subgradient methods for convex problems
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Variational Dropout Sparsifies Deep Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fast ConvNets Using Group-wise Brain Damage
• Understanding deep learning requires rethinking generalization
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Learning Multiple Layers of Features from Tiny Images

Related

• Learning Sparse Neural Networks via Sensitivity-Driven Regularization
• Weight Pruning via Adaptive Sparsity Loss
• Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures
• HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization
• Dual sparse training framework: inducing activation map sparsity via Transformed ℓ1 regularization