Time-independent Generalization Bounds for SGLD in Non-convex Settings

2021/11/25 by Tyler Farghly, Farghly, Tyler, Patrick Rebeschini +1 · 2 citations
Mathematics · Medicine · #Advanced MRI Techniques and Applications #Advanced Neuroimaging Techniques and Applications #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods #Optimization and Control (math.OC) #Probability (math.PR)

paper · pdf · doi:10.48550/arxiv.2111.12876

openalex publication_date 2021/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We establish generalization error bounds for stochastic gradient Langevin dynamics (SGLD) with constant learning rate under the assumptions of dissipativity and smoothness, a setting that has received increased attention in the sampling/optimization literature. Unlike existing bounds for SGLD in non-convex settings, ours are time-independent and decay to zero as the sample size increases. Using the framework of uniform stability, we establish time-independent bounds by exploiting the Wasserstein contraction property of the Langevin diffusion, which also allows us to circumvent the need to bound gradients using Lipschitz-like assumptions. Our analysis also supports variants of SGLD that use different discretization methods, incorporate Euclidean projections, or use non-isotropic noise.

Cited by

• Beyond Real Data: Synthetic Data through the Lens of Regularization
• Generalization of Gibbs and Langevin Monte Carlo Algorithms in the Interpolation Regime

Related

• Time-Independent Information-Theoretic Generalization Bounds for SGLD
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• A sharp uniform-in-time error estimate for Stochastic Gradient Langevin Dynamics
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a\n nonasymptotic analysis
• Generalization Bounds for Label Noise Stochastic Gradient Descent