Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

2021/08/26 by Wuyang Chen, Chen, Wuyang, Xinyu Gong +12
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2108.11939

openalex publication_date 2021/08/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Neural Architecture Search via Parameter Sharing
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• SNAS: Stochastic Neural Architecture Search
• On Exact Computation with an Infinitely Wide Neural Net
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• Predicting Neural Network Accuracy from Weights
• On the Number of Linear Regions of Convolutional Neural Networks
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Neural Architecture Search without Training
• EfficientDet: Scalable and Efficient Object Detection

Related

• Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search
• Neural Architecture Search without Training
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Revisiting Training-free NAS Metrics: An Efficient Training-based Method
• Graph is all you need? Lightweight data-agnostic neural architecture search without training