Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling

2020/06/09 by Giulio Franzese, Rosa Candela, Franzese, Giulio +7
Computer Science · Mathematics · #65C05 #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #G.3 #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #acm:65C05 #cs.LG #msc:65C05 #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.05087

arxiv created 2020/06/09 · openalex publication_date 2020/06/09 · arxiv updated 2020/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work we define a unified mathematical framework to deepen our understanding of the role of stochastic gradient (SG) noise on the behavior of Markov chain Monte Carlo sampling (SGMCMC) algorithms. Our formulation unlocks the design of a novel, practical approach to posterior sampling, which makes the SG noise isotropic using a fixed learning rate that we determine analytically, and that requires weaker assumptions than existing algorithms. In contrast, the common traits of existing \sgmcmc algorithms is to approximate the isotropy condition either by drowning the gradients in additive noise (annealing the learning rate) or by making restrictive assumptions on the \sg noise covariance and the geometry of the loss landscape. Extensive experimental validations indicate that our proposal is competitive with the state-of-the-art on \sgmcmc, while being much more practical to use.

Citations

• Gradient-based learning applied to document recognition
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Learning Multiple Layers of Features from Tiny Images

Related

• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
• Stochastic Gradient Descent as Approximate Bayesian Inference
• A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
• sgmcmc: An R Package for Stochastic Gradient Markov Chain Monte Carlo
• Improving Sampling Accuracy of Stochastic Gradient MCMC Methods via Non-uniform Subsampling of Gradients