Searching for Fast Model Families on Datacenter Accelerators

2021/02/10 by Sheng Li, Li, Sheng, Mingxing Tan +11 · 2 citations
Computer Science · Neuroscience · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2102.05610

openalex publication_date 2021/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural Architecture Search (NAS), together with model scaling, has shown remarkable progress in designing high accuracy and fast convolutional architecture families. However, as neither NAS nor model scaling considers sufficient hardware architecture details, they do not take full advantage of the emerging datacenter (DC) accelerators. In this paper, we search for fast and accurate CNN model families for efficient inference on DC accelerators. We first analyze DC accelerators and find that existing CNNs suffer from insufficient operational intensity, parallelism, and execution efficiency. These insights let us create a DC-accelerator-optimized search space, with space-to-depth, space-to-batch, hybrid fused convolution structures with vanilla and depthwise convolutions, and block-wise activation functions. On top of our DC accelerator optimized neural architecture search space, we further propose a latency-aware compound scaling (LACS), the first multi-objective compound scaling method optimizing both accuracy and latency. Our LACS discovers that network depth should grow much faster than image size and network width, which is quite different from previous compound scaling results. With the new search space and LACS, our search and scaling on datacenter accelerators results in a new model series named EfficientNet-X. EfficientNet-X is up to more than 2X faster than EfficientNet (a model series with state-of-the-art trade-off on FLOPs and accuracy) on TPUv3 and GPUv100, with comparable accuracy. EfficientNet-X is also up to 7X faster than recent RegNet and ResNeSt on TPUv3 and GPUv100.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• Designing Neural Network Architectures using Reinforcement Learning
• Enabling Sparse Winograd Convolution by Native Pruning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Searching for Activation Functions
• Progressive Neural Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Once-for-All: Train One Network and Specialize it for Efficient Deployment

Cited by

• Composer: A Search Framework for Hybrid Neural Architecture Design
• MSPT: A Lightweight Face Image Quality Assessment Method with Multi-stage Progressive Training

Related

• FLASH: Fast Neural Architecture Search with Hardware Optimization
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• GPUNet: Searching the Deployable Convolution Neural Networks for GPUs
• NASH: Neural Architecture and Accelerator Search for Multiplication-Reduced Hybrid Models
• Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces