Improving Levenberg-Marquardt Algorithm for Neural Networks

2022/12/17 by Pooladzandi, Omead, Zhou, Yiming
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2212.08769

Abstract

We explore the usage of the Levenberg-Marquardt (LM) algorithm for regression (non-linear least squares) and classification (generalized Gauss-Newton methods) tasks in neural networks. We compare the performance of the LM method with other popular first-order algorithms such as SGD and Adam, as well as other second-order algorithms such as L-BFGS , Hessian-Free and KFAC. We further speed up the LM method by using adaptive momentum, learning rate line search, and uphill step acceptance.

Related

• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networks
• Tensor-Based Foundations of Ordinary Least Squares and Neural Network Regression Models
• Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization
• mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization