The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures

2020/06/29 by Yawei Li, Li, Yawei, Wen Li +11 · 1 citation
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2006.16242

openalex publication_date 2020/06/29 · openalex created_date 2021/06/07 · openalex updated_date 2026/07/28

Abstract

In this paper, we tackle the problem of convolutional neural network design. Instead of focusing on the design of the overall architecture, we investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Based on that, we articulate the heterogeneity hypothesis: with the same training protocol, there exists a layer-wise differentiated network architecture (LW-DNA) that can outperform the original network with regular channel configurations but with a lower level of model complexity. The LW-DNA models are identified without extra computational cost or training time compared with the original network. This constraint leads to controlled experiments which direct the focus to the importance of layer-wise specific channel configurations. LW-DNA models come with advantages related to overfitting, i.e. the relative relationship between model complexity and dataset size. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration. The resultant LW-DNA models consistently outperform the baseline models. Code is available at https://github.com/ofsoundof/Heterogeneity_Hypothesis.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Gradient-based learning applied to document recognition
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DARTS: Differentiable Architecture Search
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• DHP: Differentiable Meta Pruning via HyperNetworks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression

Related

• Rethinking Layer-wise Feature Amounts in Convolutional Neural Network Architectures
• Learning Features with Parameter-Free Layers
• Conflicting Bundles: Adapting Architectures Towards the Improved\n Training of Deep Neural Networks
• About Pyramid Structure in Convolutional Neural Networks
• Learnable Heterogeneous Convolution: Learning both topology and strength