Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search

2020/06/13 by Vu Nguyen, Nguyen, Vu, Tam Le +6 · 3 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.07593

23 pages, camera ready ICML2021

openalex publication_date 2020/06/13 · openalex created_date 2020/06/19 · arxiv created 2021/06/10 · arxiv updated 2021/06/11 · openalex updated_date 2026/07/28

Abstract

Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), which is a negative definite variant of OT, we develop a novel discrepancy for neural architectures, and demonstrate it within a Gaussian process surrogate model for the sequential NAS settings. Furthermore, we derive a novel parallel NAS, using quality k-determinantal point process on the GP posterior, to select diverse and high-performing architectures from a discrete set of candidates. Empirically, we demonstrate that our TW-based approaches outperform other baselines in both sequential and parallel NAS.

Citations

• Scalable Bayesian Optimization Using Deep Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Batch Bayesian Optimization via Local Penalization
• Designing Neural Network Architectures using Reinforcement Learning
• Large-Scale Evolution of Image Classifiers
• A Tutorial on Bayesian Optimization
• Neural Architecture Optimization
• SNAS: Stochastic Neural Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Tree-Wasserstein Barycenter for Large-Scale Multilevel Clustering and Scalable Bayes
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits

Cited by

• RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design
• Spectrum Gaussian Processes Based On Tunable Basis Functions
• Tree-Sliced Wasserstein Distance with Nonlinear Projection

Related

• Traditional and accelerated gradient descent for neural architecture search
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• SNAS: Stochastic Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search