Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

2019/11/02 by Bao Wang, Wang, Bao, Difan Zou +5
Computer Science · Mathematics · Medicine · #62Dxx #65Cxx #65Yxx #Advanced Neuroimaging Techniques and Applications #Computation (stat.CO) #FOS: Computer and information sciences #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Numerical Analysis (math.NA)

paper · pdf · doi:10.48550/arxiv.1911.00782

openalex publication_date 2019/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior sampling. However, SGLD typically suffers from slow convergence rate due to its large variance caused by the stochastic gradient. In order to alleviate these drawbacks, we leverage the recently developed Laplacian Smoothing (LS) technique and propose a Laplacian smoothing stochastic gradient Langevin dynamics (LS-SGLD) algorithm. We prove that for sampling from both log-concave and non-log-concave densities, LS-SGLD achieves strictly smaller discretization error in 2-Wasserstein distance, although its mixing rate can be slightly slower. Experiments on both synthetic and real datasets verify our theoretical results, and demonstrate the superior performance of LS-SGLD on different machine learning tasks including posterior sampling, Bayesian logistic regression and training Bayesian convolutional neural networks. The code is available at \urlhttps://github.com/BaoWangMath/LS-MCMC.

Citations

• Stochastic Gradient Hamiltonian Monte Carlo
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• On the Theory of Variance Reduction for Stochastic Gradient Monte Carlo
• Stochastic Gradient Hamiltonian Monte Carlo with Variance Reduction for Bayesian Inference
• Laplacian Smoothing Gradient Descent
• DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Related

• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• The True Cost of Stochastic Gradient Langevin Dynamics
• No Free Lunch for Stochastic Gradient Langevin Dynamics
• (Non-) asymptotic properties of Stochastic Gradient Langevin Dynamics
• Langevin Monte Carlo without smoothness