Kalman Gradient Descent: Adaptive Variance Reduction in Stochastic Optimization

2018/10/29 by James Vuckovic, Vuckovic, James
Computer Science · Decision Sciences · #Stochastic Gradient Optimization Techniques #Gaussian Processes and Bayesian Inference #Advanced Bandit Algorithms Research

paper · pdf · doi:10.48550/arxiv.1810.12273

Abstract

We introduce Kalman Gradient Descent, a stochastic optimization algorithm that uses Kalman filtering to adaptively reduce gradient variance in stochastic gradient descent by filtering the gradient estimates. We present both a theoretical analysis of convergence in a non-convex setting and experimental results which demonstrate improved performance on a variety of machine learning areas including neural networks and black box variational inference. We also present a distributed version of our algorithm that enables large-dimensional optimization, and we extend our algorithm to SGD with momentum and RMSProp.

Citations

• Black Box Variational Inference

Related

• Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
• Gradient Descent, Stochastic Optimization, and Other Tales
• Variance-Reduced Methods for Machine Learning
• Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction
• Variance Reduction for Faster Non-Convex Optimization