SVRG and Beyond via Posterior Correction

2025/12/01 by Nico Daheim, Daheim, Nico, Thomas Möllenhoff +5
Computer Science · #Artificial Intelligence (cs.AI) #Bayes' theorem #Bayesian probability #Extension (predicate logic) #FOS: Computer and information sciences #Gaussian #Generative Adversarial Networks and Image Synthesis #Gradient descent #Hessian matrix #Machine Learning (cs.LG) #Natural Language Processing Techniques #Posterior probability #Topic Modeling

paper · open access · doi:10.48550/arxiv.2512.01930

published in TUbilio (Technical University of Darmstadt) (Technische Universität Darmstadt)

openalex publication_date 2025/12/01 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections. Originally proposed over a decade ago, these methods have never been connected to any Bayesian method at a fundamental level. Here, we fill this gap and derive surprising new connections of SVRG to a recently proposed Bayesian method called `posterior correction'. Our main contribution is to show that SVRG can be recovered as a special case of posterior-correction over isotropic-Gaussian posteriors. Novel extensions of SVRG are automatically obtained by using more flexible exponential-family posteriors. We derive two new such extensions by using Gaussian families: a Newton-like variant with novel Hessian corrections, and an Adam-like extension that scales to large problems. Our work is the first to connect SVRG to Bayes and use it to speed-up training.

Citations

• Knowledge Adaptation as Posterior Correction
• SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learning
• The Llama 3 Herd of Models
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients
• Variational Learning is Effective for Large Deep Networks
• A Coefficient Makes SVRG Effective
• Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
• Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
• Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Variance Reduction in Deep Learning: More Momentum is All You Need
• Training Verifiers to Solve Math Word Problems
• SVRG Meets AdaGrad: Painless Variance Reduction
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Training Binary Neural Networks using the Bayesian Learning Rule
• Handling the Positive-Definite Constraint in the Bayesian Learning Rule
• Reducing the variance in online optimization by transporting past gradients
• Momentum-Based Variance Reduction in Non-Convex SGD
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• Quasi-hyperbolic momentum and Adam for deep learning
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
• Advances in Variational Inference
• Tracking the gradients using the Hessian: A new look at variance reducing stochastic methods
• Non-convex Finite-Sum Optimization Via SCSG Methods
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• Stop Wasting My Gradients: Practical SVRG
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Smoothed Gradients for Stochastic Variational Inference
• Black Box Variational Inference
• Describing Textures in the Wild
• Minimizing Finite Sums with the Stochastic Average Gradient
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
• Expectation Propagation for approximate Bayesian inference

Related

• A Coefficient Makes SVRG Effective
• On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
• Stop Wasting My Gradients: Practical SVRG
• Towards Better Generalization: BP-SVRG in Training Deep Neural Networks
• VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning