2020/09/06 by Tuyen Trung Truong, Hang-Tuan Nguyen · 1 citation
Computer Science · Engineering · Mathematics · #Stochastic Gradient Optimization Techniques #Sparse and Compressive Sensing Techniques #Advanced Neural Network Applications #Backtracking #Gradient descent #Algorithm #Convergence (economics) #Look-ahead #Mathematical optimization #Mathematics #Lipschitz continuity #Computer science #Artificial intelligence #Artificial neural network
paper · pdf · doi:10.1007/s00245-020-09718-8
openalex publication_date 2020/09/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Abstract In this paper, we provide new results and algorithms (including backtracking versions of Nesterov accelerated gradient and Momentum) which are more applicable to large scale optimisation as in Deep Neural Networks. We also demonstrate that Backtracking Gradient Descent (Backtracking GD) can obtain good upper bound estimates for local Lipschitz constants for the gradient, and that the convergence rate of Backtracking GD is similar to that in classical work of Armijo. Experiments with datasets CIFAR10 and CIFAR100 on various popular architectures verify a heuristic argument that Backtracking GD stabilises to a finite union of sequences constructed from Standard GD for the mini-batch practice, and show that our new algorithms (while automatically fine tuning learning rates) perform better than current state-of-the-art methods such as Adam, Adagrad, Adadelta, RMSProp, Momentum and Nesterov accelerated gradient. To help readers avoiding the confusion between heuristics and more rigorously justified algorithms, we also provide a review of the current state of convergence results for gradient descent methods. Accompanying source codes are available on GitHub.