A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

2020/10/19 by Wei Deng, Guang Lin, Deng, Wei +3 · 3 citations
Computer Science · Mathematics · #Computation (stat.CO) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Markov Chains and Monte Carlo Methods

paper · pdf · doi:10.48550/arxiv.2010.09800

openalex publication_date 2020/10/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a scalable dynamic importance sampler, which automatically flattens the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a \it unique fixed-point, regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority to avoid the local trap problem in training deep neural networks.

Citations

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural\n Networks
• A Stochastic Approximation Method
• Stochastic Gradient Hamiltonian Monte Carlo
• Random Erasing Data Augmentation
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
• Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC\n via Variance Reduction

Cited by

• Opt3DGS: Optimizing 3D Gaussian Splatting with Adaptive Exploration and Curvature-Aware Exploitation
• Flatness-Aware Stochastic Gradient Langevin Dynamics
• An Evolutionary Multi-objective Optimization for Replica-Exchange-based Physics-informed Operator Learning Network

Related

• Interacting Contour Stochastic Gradient Langevin Dynamics
• Scalable Natural Gradient Langevin Dynamics in Practice
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• Federated Stochastic Gradient Langevin Dynamics
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts