DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration

2019/12/21 by Wenxiao Wang, Shuai Zhao, Wang, Wenxiao +9 · 3 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.1912.10178

9 pages, 5 figures

arxiv created 2019/12/21 · openalex publication_date 2019/12/21 · arxiv updated 2019/12/24 · openalex created_date 2019/12/26 · openalex updated_date 2026/07/28

Abstract

Neural network pruning is one of the most popular methods of accelerating the inference of deep convolutional neural networks (CNNs). The dominant pruning methods, filter-level pruning methods, evaluate their performance through the reduction ratio of computations and deem that a higher reduction ratio of computations is equivalent to a higher acceleration ratio in terms of inference time. However, we argue that they are not equivalent if parallel computing is considered. Given that filter-level pruning only prunes filters in layers and computations in a layer usually run in parallel, most computations reduced by filter-level pruning usually run in parallel with the un-reduced ones. Thus, the acceleration ratio of filter-level pruning is limited. To get a higher acceleration ratio, it is better to prune redundant layers because computations of different layers cannot run in parallel. In this paper, we propose our Discrimination based Block-level Pruning method (DBP). Specifically, DBP takes a sequence of consecutive layers (e.g., Conv-BN-ReLu) as a block and removes redundant blocks according to the discrimination of their output features. As a result, DBP achieves a considerable acceleration ratio by reducing the depth of CNNs. Extensive experiments show that DBP has surpassed state-of-the-art filter-level pruning methods in both accuracy and acceleration ratio. Our code will be made available soon.

Citations

• Distilling the Knowledge in a Neural Network
• ImageNet Large Scale Visual Recognition Challenge
• Do Deep Nets Really Need to be Deep?
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Learning Efficient Convolutional Networks through Network Slimming
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning
• Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Online Training and Pruning of Deep Reinforcement Learning Networks
• IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning
• FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition

Related

• Structured Pruning for Deep Convolutional Neural Networks: A Survey
• Partition Pruning: Parallelization-Aware Pruning for Deep Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Pruning Filters for Efficient ConvNets
• RL-Pruner: Structured Pruning Using Reinforcement Learning for CNN Compression and Acceleration