2020/01/31 by Santiago González, Risto Miikkulainen, Gonzalez, Santiago +1
Computer Science · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Metaheuristic Optimization Algorithms Research #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.2002.00059
openalex publication_date 2020/01/31 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Metalearning of deep neural network (DNN) architectures and hyperparameters\nhas become an increasingly important area of research. Loss functions are a\ntype of metaknowledge that is crucial to effective training of DNNs, however,\ntheir potential role in metalearning has not yet been fully explored. Whereas\nearly work focused on genetic programming (GP) on tree representations, this\npaper proposes continuous CMA-ES optimization of multivariate Taylor polynomial\nparameterizations. This approach, TaylorGLO, makes it possible to represent and\nsearch useful loss functions more effectively. In MNIST, CIFAR-10, and SVHN\nbenchmark tasks, TaylorGLO finds new loss functions that outperform functions\npreviously discovered through GP, as well as the standard cross-entropy loss,\nin fewer generations. These functions serve to regularize the learning task by\ndiscouraging overfitting to the labels, which is particularly useful in tasks\nwhere limited training data is available. The results thus demonstrate that\nloss function optimization is a productive new avenue for metalearning.\n