Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks

2018/10/15 by Xiaodong Cui, Wěi Zhāng, Cui, Xiaodong +6 · 3 citations
Computer Science · #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Metaheuristic Optimization Algorithms Research #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1810.06773

openalex publication_date 2018/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve the average fitness of the population. With a back-off strategy in the SGD step and an elitist strategy in the evolution step, it guarantees that the best fitness in the population will never degrade. In addition, individuals in the population optimized with various SGD-based optimizers using distinct hyper-parameters in the SGD step are considered as competing species in a coevolution setting such that the complementarity of the optimizers is also taken into account. The effectiveness of ESGD is demonstrated across multiple applications including speech recognition, image recognition and language modeling, using networks with a variety of deep architectures.

Cited by

• BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
• Evolution imposes an inductive bias that alters and accelerates learning dynamics
• Demystifying Parallel and Distributed Deep Learning

Related

• Optimizing Deep Neural Networks through Neuroevolution with Stochastic Gradient Descent
• Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks
• Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale
• Variational Stochastic Gradient Descent for Deep Neural Networks
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks