2021/06/30 by Ghassen Jerfel, Jerfel, Ghassen, Serena Wang +11 · 6 citations
Computer Science · Engineering · Mathematics · #Computation (stat.CO) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #cs.LG #stat.CO #stat.ML
paper · pdf · doi:10.48550/arxiv.2106.15980
Accepted for the 37th Conference on Uncertainty in Artificial Intelligence (UAI 2021)
arxiv created 2021/06/30 · openalex publication_date 2021/06/30 · arxiv updated 2021/07/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Variational Inference (VI) is a popular alternative to asymptotically exact sampling in Bayesian inference. Its main workhorse is optimization over a reverse Kullback-Leibler divergence (RKL), which typically underestimates the tail of the posterior leading to miscalibration and potential degeneracy. Importance sampling (IS), on the other hand, is often used to fine-tune and de-bias the estimates of approximate Bayesian inference procedures. The quality of IS crucially depends on the choice of the proposal distribution. Ideally, the proposal distribution has heavier tails than the target, which is rarely achievable by minimizing the RKL. We thus propose a novel combination of optimization and sampling techniques for approximate Bayesian inference by constructing an IS proposal distribution through the minimization of a forward KL (FKL) divergence. This approach guarantees asymptotic consistency and a fast convergence towards both the optimal IS estimator and the optimal variational approximation. We empirically demonstrate on real data that our method is competitive with variational boosting and MCMC.