Progressive Neural Architecture Search

2017/12/02 by Chenxi Liu, Barret Zoph, Liu, Chenxi +17 · 1 voice · 43 citations
Computer Science · #Advanced Neural Network Applications #Machine Learning and Data Classification #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.1712.00559

Abstract

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

Citations

• Practical Bayesian Optimization of Machine Learning Algorithms
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Densely Connected Convolutional Networks
• Neural Architecture Search with Reinforcement Learning
• DeepArchitect: Automatically Designing and Training Deep Architectures
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Efficient Architecture Search by Network Transformation

Cited by

• ReCowGnition: A Realistic Biometric Benchmark for Cow Face Recognition
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• Quantum Decision Transformers (QDT): Synergistic Entanglement and Interference for Offline Reinforcement Learning
• Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
• MCAQ-YOLO: Morphological Complexity-Aware Quantization for Efficient Object Detection with Curriculum Learning
• Beyond Deceptive Flatness: Dual-Order Solution for Strengthening Adversarial Transferability
• SNAS: Stochastic Neural Architecture Search
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Exploiting the Potential of Standard Convolutional Autoencoders for\n Image Restoration by Evolutionary Search
• Interaction-aware Factorization Machines for Recommender Systems
• Fine-Grained Neural Architecture Search
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Feature Partitioning for Efficient Multi-Task Architectures
• FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Data
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Spiking Neural Network Architecture Search: A Survey
• Optimally Deep Networks -- Adapting Model Depth to Datasets for Superior Efficiency
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks
• WeatherCycle: Unpaired Multi-Weather Restoration via Color Space Decoupled Cycle Learning
• Neural Architecture Search in Embedding Space
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• Auto-Meta: Automated Gradient Based Meta Learner Search
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Network Decoupling: From Regular to Depthwise Separable Convolutions
• NAT: Learning to Attack Neurons for Enhanced Adversarial Transferability
• A Continuous Encoding-Based Representation for Efficient Multi-Fidelity Multi-Objective Neural Architecture Search
• Deep Learning in Mobile and Wireless Networking: A Survey
• IsoSched: Preemptive Tile Cascaded Scheduling of Multi-DNN via Subgraph Isomorphism
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Language Models with Transformers
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Understanding Neural Architecture Search Techniques
• Graph Lineages and Skeletal Graph Products
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• EA-ViT: Efficient Adaptation for Elastic Vision Transformer
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation

Discussions

• Progressive Neural Architecture Search [hn, 45 points, 0 comments]

Related

• Efficient Progressive Neural Architecture Search
• A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Effective Building Block Design for Deep Convolutional Neural Networks using Search
• Stage-Wise Neural Architecture Search