BN-NAS: Neural Architecture Search with Batch Normalization

2021/08/16 by Boyu Chen, Peixia Li, Chen, Boyu +14 · 2 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Architecture #Artificial intelligence #Artificial neural network #Computer Vision and Pattern Recognition (cs.CV) #Computer network #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine learning #Network architecture #Normalization (sociology) #Subnet #Training (meteorology) #Training set #cs.CV

paper · pdf · doi:10.48550/arxiv.2108.07375

published in arXiv (Cornell University) (Cornell University) · ICCV 2021

arxiv created 2021/08/16 · openalex publication_date 2021/08/16 · arxiv updated 2021/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Learning Efficient Convolutional Networks through Network Slimming
• Efficient Neural Architecture Search via Parameter Sharing
• DARTS: Differentiable Architecture Search
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• EfficientDet: Scalable and Efficient Object Detection
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature
• From Tiny Machine Learning to Tiny Deep Learning: A Survey

Related

• Training BatchNorm Only in Neural Architecture Search and Beyond
• NAS-BNN: Neural Architecture Search for Binary Neural Networks
• BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search
• BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics