Progressive Neural Architecture Search

2017/12/02 by Chenxi Liu, Barret Zoph, Liu, Chenxi +17 · 1 voice · 167 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Algorithm #Artificial intelligence #Artificial neural network #Computer science #Convolutional neural network #Machine Learning and Data Classification #Machine learning #Mathematics #Reinforcement learning #Space (punctuation) #State (computer science) #State space #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1712.00559

published in arXiv (Cornell University) (Cornell University) · To appear in ECCV 2018 as oral. The code and checkpoint for PNASNet-5 trained on ImageNet (both Mobile and Large) can now be downloaded from https://github.com/tensorflow/models/tree/master/research/slim#Pretrained. Also see https://github.com/chenxi116/PNASNet.TF for refactored and simplified TensorFlow code; see https://github.com/chenxi116/PNASNet.pytorch for exact conversion to PyTorch

openalex publication_date 2017/12/02 · arxiv created 2018/07/26 · arxiv updated 2018/07/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

Citations

• Practical Bayesian Optimization of Machine Learning Algorithms
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Densely Connected Convolutional Networks
• Neural Architecture Search with Reinforcement Learning
• DeepArchitect: Automatically Designing and Training Deep Architectures
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Efficient Architecture Search by Network Transformation

Cited by

• ReCowGnition: A Realistic Biometric Benchmark for Cow Face Recognition
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• Quantum Decision Transformers (QDT): Synergistic Entanglement and Interference for Offline Reinforcement Learning
• Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
• MCAQ-YOLO: Morphological Complexity-Aware Quantization for Efficient Object Detection with Curriculum Learning
• Beyond Deceptive Flatness: Dual-Order Solution for Strengthening Adversarial Transferability
• SNAS: Stochastic Neural Architecture Search
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search
• Interaction-aware Factorization Machines for Recommender Systems
• Fine-Grained Neural Architecture Search
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Feature Partitioning for Efficient Multi-Task Architectures
• FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Spiking Neural Network Architecture Search: A Survey
• Optimally Deep Networks -- Adapting Model Depth to Datasets for Superior Efficiency
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks
• WeatherCycle: Unpaired Multi-Weather Restoration via Color Space Decoupled Cycle Learning
• Neural Architecture Search in Embedding Space
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• Auto-Meta: Automated Gradient Based Meta Learner Search
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Network Decoupling: From Regular to Depthwise Separable Convolutions
• NAT: Learning to Attack Neurons for Enhanced Adversarial Transferability
• A Continuous Encoding-Based Representation for Efficient Multi-Fidelity Multi-Objective Neural Architecture Search
• Deep Learning in Mobile and Wireless Networking: A Survey
• IsoSched: Preemptive Tile Cascaded Scheduling of Multi-DNN via Subgraph Isomorphism
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Language Models with Transformers
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Understanding Neural Architecture Search Techniques
• Knowledge-aware Diffusion-Enhanced Multimedia Recommendation
• Graph Lineages and Skeletal Graph Products
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• EA-ViT: Efficient Adaptation for Elastic Vision Transformer
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• ACME: Adaptive Customization of Large Models via Distributed Systems
• Design and Implementation of an Annotation-Driven Drone Autonomy Tool Using YOLOv8–V11 Architectures for Real-Time Object Detection and Distance Estimation
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods
• Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• Text Embedding Knows How to Quantize Text-Guided Diffusion Models
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Zero-Shot Neural Architecture Search with Weighted Response Correlation
• DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Semi-Supervised Neural Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• NN-Former: Rethinking Graph Structure in Neural Architecture Representation
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Deep Electromagnetic Structure Design Under Limited Evaluation Budgets
• V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation
• HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning
• Structure-Aware Automatic Channel Pruning by Searching with Graph Embedding
• Path-Level Network Transformation for Efficient Architecture Search
• Loss Functions for Predictor-based Neural Architecture Search
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Adversarially Robust Neural Architectures
• RePr: Improved Training of Convolutional Filters
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Global optimization of graph acquisition functions for neural architecture search
• DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search
• Searching for Fast Model Families on Datacenter Accelerators
• Evolving Rewards to Automate Reinforcement Learning
• NASIB: Neural Architecture Search withIn Budget
• Composite Binary Decomposition Networks
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• MLFriend: Interactive Prediction Task Recommendation for Event-Driven Time-Series Data
• DarwinML: A Graph-based Evolutionary Algorithm for Automated Machine Learning
• L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers
• AGAN: Towards Automated Design of Generative Adversarial Networks
• Layout Design for Intelligent Warehouse by Evolution with Fitness Approximation
• Demystifying Parallel and Distributed Deep Learning
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• MedNNS: Supernet-based Medical Task-Adaptive Neural Network Search
• Could AI Trace and Explain the Origins of AI-Generated Images and Text?
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures
• Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition

Discussions

• Progressive Neural Architecture Search [hn, 45 points, 0 comments]

Related

• Efficient Progressive Neural Architecture Search
• A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Effective Building Block Design for Deep Convolutional Neural Networks using Search
• Stage-Wise Neural Architecture Search