Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization

2021/02/13 by Wissam J. Baddar, Baddar, Wissam J., Seungju Han +5
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Video Surveillance and Tracking Methods

paper · pdf · doi:10.48550/arxiv.2102.06870

openalex publication_date 2021/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose self-reorganizing and rejuvenating convolutional neural networks; a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The reorganized parameters are clustered to avoid parameter redundancies. As such, redundant neurons with similar activations are merged leaving room for the remaining parameters to rejuvenate. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters. As a result, the network capacity utilization increases improving the baseline network performance without any changes to the network structure. The proposed method can be applied to various network architectures during the training stage, or applied to a pre-trained model improving its performance. Experimental results showed that the proposed method is model-agnostic and can be applied to any backbone architecture increasing its performance due to the elevated utilization of the network capacity.

Citations

• Learning to Segment Object Candidates
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How Does the Brain Solve Visual Object Recognition?
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Pruning Filters for Efficient ConvNets
• Neural Architecture Search with Reinforcement Learning
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• YOLOv3: An Incremental Improvement
• Efficient Architecture Search by Network Transformation
• Brain-Like Object Recognition with High-Performing Shallow Recurrent\n ANNs

Related

• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Self Expanding Convolutional Neural Networks
• Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules
• Combining Neuro-Evolution of Augmenting Topologies with Convolutional Neural Networks
• Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network