Towards Compact CNNs via Collaborative Compression

2021/05/24 by Yuchao Li, Li, Yuchao, Shaohui Lin +17 · 3 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2105.11228

openalex publication_date 2021/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Channel pruning and tensor decomposition have received extensive attention in convolutional neural network compression. However, these two techniques are traditionally deployed in an isolated manner, leading to significant accuracy drop when pursuing high compression rates. In this paper, we propose a Collaborative Compression (CC) scheme, which joints channel pruning and tensor decomposition to compress CNN models by simultaneously learning the model sparsity and low-rankness. Specifically, we first investigate the compression sensitivity of each layer in the network, and then propose a Global Compression Rate Optimization that transforms the decision problem of compression rate into an optimization problem. After that, we propose multi-step heuristic compression to remove redundant compression units step-by-step, which fully considers the effect of the remaining compression space (i.e., unremoved compression units). Our method demonstrates superior performance gains over previous ones on various datasets and backbone architectures. For example, we achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet classification with deep convolutional neural networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

Cited by

• CAMP-HiVe: Cyclic Pair Merging based Efficient DNN Pruning with Hessian-Vector Approximation for Resource-Constrained Systems
• FGFP: A Fractional Gaussian Filter and Pruning for Deep Neural Networks Compression
• CoSeP: Complementary Separability Pruning via Class-Separability Clustering

Related

• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning
• Tensor Reordering for CNN Compression
• Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization