A note on the unique properties of the Kullback--Leibler divergence for sampling via gradient flows

2025/07/06 by Crucinio, Francesca Romana · 1 citation
#Computation (stat.CO) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Methodology (stat.ME) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2507.04330

Abstract

We consider the problem of sampling from a probability distribution π. It is well known that this can be written as an optimisation problem over the space of probability distribution in which we aim to minimise a divergence from π. and The optimisation problem is normally solved through gradient flows in the space of probability distribution with an appropriate metric. We show that the Kullback--Leibler divergence is the only divergence in the family of Bregman divergences whose gradient flow w.r.t. many popular metrics does not require knowledge of the normalising constant of π.

Citations

• Sequential Monte Carlo approximations of Wasserstein--Fisher--Rao gradient flows
• Weighted quantization using MMD: From mean field to mean shift via gradient flows
• Inclusive KL Minimization: A Wasserstein-Fisher-Rao Gradient Flow Perspective
• Provable Convergence and Limitations of Geometric Tempering for Langevin Dynamics
• Stein transport for Bayesian inference
• Fisher-Rao Gradient Flow: Geodesic Convexity and Functional Inequalities
• Sampling in Unit Time with Kernel Fisher-Rao Flow
• A connection between Tempering and Entropic Mirror Descent
• Sampling via Gradient Flows in the Space of Probability Measures
• An Explicit Expansion of the Kullback-Leibler Divergence along its Fisher-Rao Gradient Flow
• Learning Gaussian Mixtures Using the Wasserstein-Fisher-Rao Gradient Flow
• Sampling with Mollified Interaction Energy Descent
• Discrete Langevin Sampler via Wasserstein Gradient Flow
• Variational inference via Wasserstein gradient flows
• Adaptive Importance Sampling meets Mirror Descent: a Bias-variance\n tradeoff
• Kernel Stein Discrepancy Descent
• SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence
• On the geometry of Stein variational gradient descent
• Maximum Mean Discrepancy Gradient Flow
• Accelerating Langevin Sampling with Birth-death
• Alpha-Beta Divergence For Variational Inference
• Analysis of Langevin Monte Carlo via convex optimization
• Stein Variational Gradient Descent as Gradient Flow
• Measuring Sample Quality with Kernels
• A JKO splitting scheme for Kantorovich-Fisher-Rao gradient flows
• Variational Inference: A Review for Statisticians
• Provable Bayesian Inference via Particle Mirror Descent
• Sequential monte carlo samplers
• Natural Gradient Works Efficiently in Learning

Cited by

• Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions

Related

• Sampling via Gradient Flows in the Space of Probability Measures
• Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance
• Sequential Monte Carlo approximations of Wasserstein--Fisher--Rao gradient flows
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint\n Support
• An Explicit Expansion of the Kullback-Leibler Divergence along its Fisher-Rao Gradient Flow