Channel Pruning for Accelerating Very Deep Neural Networks

2017/07/19 by Yihui He, Xiangyu Zhang, He, Yihui +3 · 90 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.1707.06168

To be appear at ICCV 2017

openalex publication_date 2017/07/19 · arxiv created 2017/08/21 · arxiv updated 2017/08/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks.Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhance the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5x speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% accuracy loss under 2x speed-up respectively, which is significant. Code has been made publicly available.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradient-based learning applied to document recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Speeding-up Convolutional Neural Networks Using Fine-tuned\n CP-Decomposition
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Fast ConvNets Using Group-wise Brain Damage
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Xception: Deep Learning with Depthwise Separable Convolutions
• Structured Pruning of Deep Convolutional Neural Networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Masking Teacher and Reinforcing Student for Distilling Vision-Language Models
• Dictionary-Based Contrastive Learning for GNSS Jamming Detection
• BTKD++: Beyond Teachers by Critically Distilling Knowledge from Teacher’s Bias
• SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot
• Change-of-Basis Pruning via Rotational Invariance
• Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity
• Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
• MI-to-Mid Distilled Compression (M2M-DC): An Hybrid-Information-Guided-Block Pruning with Progressive Inner Slicing Approach to Model Compression
• Lipschitz-aware Linearity Grafting for Certified Robustness
• TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples
• Convergence, design and training of continuous-time dropout as a random batch method
• MC#: Mixture Compressor for Mixture-of-Experts Large Models
• PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
• Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
• StructPrune: Structured Global Pruning asymptotics with O(√(N)) GPU Memory
• GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation
• Convolutional Neural Network Optimization for Beehive Classification Using Bioacoustic Signals
• ResidualViT for Efficient Temporally Dense Video Encoding
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Formal Algorithms for Model Efficiency
• ONG: One-Shot NMF-based Gradient Masking for Efficient Model Sparsification
• HALO: Learning to Prune Neural Networks with Shrinkage
• Recurrent Residual Module for Fast Inference in Videos
• Channel-Level Variable Quantization Network for Deep Image Compression
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• CNN Acceleration by Low-rank Approximation with Quantized Factors
• Task complexity shapes internal representations and robustness in neural networks
• V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Models
• FAIR-Pruner: Leveraging Tolerance of Difference for Flexible Automatic Layer-Wise Neural Network Pruning
• FedAPTA: Federated Multi-task Learning for Heterogeneous Devices with Adaptive Layer-wise Pruning and Task-aware Aggregation
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Unifying and Merging Well-trained Deep Neural Networks for Inference Stage
• EA-ViT: Efficient Adaptation for Elastic Vision Transformer
• The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models
• Efficient Neural Network Compression
• Deep Mixture of Experts via Shallow Embedding
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Artificial Intelligence-Enhanced Wearable Blood Pressure Monitoring in Resource-Limited Settings: A Co-Design of Sensors, Model, and Deployment
• Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• Feature Statistics Guided Efficient Filter Pruning
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training
• SBNet: Sparse Blocks Network for Fast Inference
• CNNPruner: Pruning Convolutional Neural Networks with Visual Analytics
• QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models
• IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning
• Compact Global Descriptor for Neural Networks
• Model Compression using Progressive Channel Pruning
• Towards Universal & Efficient Model Compression via Exponential Torque Pruning
• ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks
• A geometric framework for momentum-based optimizers for low-rank training
• Structure-Aware Automatic Channel Pruning by Searching with Graph Embedding
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Writer-Aware CNN for Parsimonious HMM-Based Offline Handwritten Chinese Text Recognition
• A novel channel pruning method for deep neural network compression
• Pruning via Iterative Ranking of Sensitivity Statistics
• CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
• CondenseNet V2: Sparse Feature Reactivation for Deep Networks
• Smooth Model Compression without Fine-Tuning
• TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks
• DAC: Data-free Automatic Acceleration of Convolutional Networks
• FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition
• Meta Pruning via Graph Metanetworks : A Universal Meta Learning Framework for Network Pruning
• Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation
• Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Models
• Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Searching for Low-Bit Weights in Quantized Neural Networks
• PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution
• SepPrune: Structured Pruning for Efficient Deep Speech Separation
• MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection
• Adversarially Robust Spiking Neural Networks with Sparse Connectivity
• CAE-ADMM: Implicit Bitrate Optimization via ADMM-based Pruning in Compressive Autoencoders
• Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogate
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks
• Optimizing Deep Neural Networks using Safety-Guided Self Compression
• How is Contrast Encoded in Deep Neural Networks?
• DNAD: Differentiable Neural Architecture Distillation
• Neuron Merging: Compensating for Pruned Neurons
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Demystifying Neural Network Filter Pruning
• CUT: Pruning Pre-Trained Multi-Task Models into Compact Models for Edge Devices
• Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Training
• Dynamic Vision Mamba

Related

• Pruning Very Deep Neural Network Channels for Efficient Inference
• Layer Pruning for Accelerating Very Deep Neural Networks
• PruneNet: Channel Pruning via Global Importance
• UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks