Full-Stack Filters to Build Minimum Viable CNNs

2019/08/06 by Kai Han, Yunhe Wang, Han, Kai +9
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Video Surveillance and Tracking Methods #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.1908.02023

Tech report

arxiv created 2019/08/06 · openalex publication_date 2019/08/06 · arxiv updated 2019/08/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep convolutional neural networks (CNNs) are usually over-parameterized, which cannot be easily deployed on edge devices such as mobile phones and smart cameras. Existing works used to decrease the number or size of requested convolution filters for a minimum viable CNN on edge devices. In contrast, this paper introduces filters that are full-stack and can be used to generate many more sub-filters. Weights of these sub-filters are inherited from full-stack filters with the help of different binary masks. Orthogonal constraints are applied over binary masks to decrease their correlation and promote the diversity of generated sub-filters. To preserve the same volume of output feature maps, we can naturally reduce the number of established filters by only maintaining a few full-stack filters and a set of binary masks. We also conduct theoretical analysis on the memory cost and an efficient implementation is introduced for the convolution of the proposed filters. Experiments on several benchmark datasets and CNN models demonstrate that the proposed method is able to construct minimum viable convolution networks of comparable performance.

Citations

• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• Gradient-based learning applied to document recognition
• Provable Bounds for Learning Some Deep Representations
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Constructing Fast Network through Deconstruction of Convolution
• ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Learning Multiple Layers of Features from Tiny Images

Related

• Compressing Deep Convolutional Neural Networks by Stacking Low-dimensional Binary Convolution Filters
• SMOF: Squeezing More Out of Filters Yields Hardware-Friendly CNN Pruning
• ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
• Compressing Deep CNNs using Basis Representation and Spectral Fine-tuning
• Training Efficient CNNS: Tweaking the Nuts and Bolts of Neural Networks for Lighter, Faster and Robust Models