Preconditioned Stochastic Gradient Descent

2015/12/14 by Xi-Lin Li · 1 voice · 109 citations
Computer Science · Engineering · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Artificial neural network #Computer science #Convergence (economics) #Gradient descent #Hessian matrix #Iterative method #Machine Learning and ELM #Mathematical optimization #Mathematics #Preconditioner #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Stochastic gradient descent #cs.LG #stat.ML

paper · pdf · doi:10.1109/tnnls.2017.2672978

published in IEEE Transactions on Neural Networks and Learning Systems 29(5), 1454-1466 (Institute of Electrical and Electronics Engineers) · 13 pages, 9 figures. To appear in IEEE Transactions on Neural Networks and Learning Systems. Supplemental materials on https://sites.google.com/site/lixilinx/home/psgd

arxiv published 2015/12/14 · arxiv created 2017/02/22 · arxiv updated 2017/02/23 · openalex publication_date 2017/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in significantly more complicated methods than SGD. This paper proposes a new method to estimate a preconditioner such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to Newton method for deterministic optimization. Unlike the preconditioners based on secant equation fitting as done in deterministic quasi-Newton methods, which assume positive definite Hessian and approximate its inverse, the new preconditioner works equally well for both convex and non-convex optimizations with exact or noisy gradients. When stochastic gradient is used, it can naturally damp the gradient noise to stabilize SGD. Efficient preconditioner estimation methods are developed, and with reasonable simplifications, they are applicable to large scaled problems. Experimental results demonstrate that equipped with the new preconditioner, without any tuning effort, preconditioned SGD can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long term memories.

Citations

• Learning representations by back-propagating errors
• Equilibrated adaptive learning rates for non-convex optimization
• Parallel training of DNNs with Natural Gradient and Parameter Averaging
• Natural Gradient Works Efficiently in Learning
• Long Short-Term Memory
• Gradient-based learning applied to document recognition
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Recurrent neural network training with preconditioned stochastic gradient descent

Cited by

• Score-Based Martingale Posteriors for Deep Neural Networks
• Martingale Posteriors for Discretely Observed Diffusions
• TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training

Discussions

• PSGD is my go-to paper whenever I think about what second-order optimization for neural networks should look like. PSGD defines a criterion that explicitly accounts for gradient noise. Someday, I’d lo [bsky, 1 points, 0 comments]

Related

• Online Second Order Methods for Non-Convex Stochastic Optimizations
• Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
• Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
• Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis
• Convergence analysis of stochastic gradient descent with adaptive preconditioning for non-convex and convex functions