2019/04/19 by Jiawei Zhang, Zhang, Jiawei · 1 citation
Computer Science · Engineering · #Advanced Multi-Objective Optimization Algorithms #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Infrared Target Detection Methodologies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Sparse and Compressive Sensing Techniques
paper · pdf · doi:10.48550/arxiv.1904.09365
openalex publication_date 2019/04/19 · openalex created_date 2022/07/29 · openalex updated_date 2026/07/28
In this paper, we will provide an introduction to the derivative-free\noptimization algorithms which can be potentially applied to train deep learning\nmodels. Existing deep learning model training is mostly based on the back\npropagation algorithm, which updates the model variables layers by layers with\nthe gradient descent algorithm or its variants. However, the objective\nfunctions of deep learning models to be optimized are usually non-convex and\nthe gradient descent algorithms based on the first-order derivative can get\nstuck into the local optima very easily. To resolve such a problem, various\nlocal or global optimization algorithms have been proposed, which can help\nimprove the training of deep learning models greatly. The representative\nexamples include the Bayesian methods, Shubert-Piyavskii algorithm, Direct,\nLIPO, MCS, GA, SCE, DE, PSO, ES, CMA-ES, hill climbing and simulated annealing,\netc. One part of these algorithms will be introduced in this paper (including\nthe Bayesian method and Lipschitzian approaches, e.g., Shubert-Piyavskii\nalgorithm, Direct, LIPO and MCS), and the remaining algorithms (including the\npopulation based optimization algorithms, e.g., GA, SCE, DE, PSO, ES and\nCMA-ES, and random search algorithms, e.g., hill climbing and simulated\nannealing) will be introduced in the follow-up paper [18] in detail.\n