Traditional and accelerated gradient descent for neural architecture search

2020/06/26 by Trillos, Nicolas Garcia, Morales, Felix, Morales, Javier
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Numerical Analysis (math.NA)

paper · doi:10.48550/arxiv.2006.15218

Abstract

In this paper we introduce two algorithms for neural architecture search (NASGD and NASAGD) following the theoretical work by two of the authors [5] which used the geometric structure of optimal transport to introduce the conceptual basis for new notions of traditional and accelerated gradient descent algorithms for the optimization of a function on a semi-discrete space. Our algorithms, which use the network morphism framework introduced in [2] as a baseline, can analyze forty times as many architectures as the hill climbing methods [2, 14] while using the same computational resources and time and achieving comparable levels of accuracy. For example, using NASGD on CIFAR-10, our method designs and trains networks with an error rate of 4.06 in only 12 hours on a single GPU.

Related

• Searching for A Robust Neural Architecture in Four GPU Hours
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• Multi-objective Neural Architecture Search with Almost No Training