Stein Variational Gaussian Processes

2020/09/25 by Thomas Pinder, Christopher Nemeth, Pinder, Thomas +4 · 3 citations
Computer Science · Environmental Science · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Artificial neural network #Bayesian inference #Bayesian probability #Benchmark (surveying) #Computer science #FOS: Computer and information sciences #Gaussian #Gaussian Processes and Bayesian Inference #Gaussian process #Gradient descent #Groundwater flow and contamination studies #Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov chain Monte Carlo #Mathematical optimization #Mathematics #Parametric statistics #Physics #Statistics #Target Tracking and Data Fusion in Sensor Networks #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2009.12141

published in arXiv (Cornell University) (Cornell University) · 26 pages, 5 figures

openalex publication_date 2020/09/25 · arxiv created 2022/01/19 · arxiv updated 2022/01/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We show how to use Stein variational gradient descent (SVGD) to carry out inference in Gaussian process (GP) models with non-Gaussian likelihoods and large data volumes. Markov chain Monte Carlo (MCMC) is extremely computationally intensive for these situations, but the parametric assumptions required for efficient variational inference (VI) result in incorrect inference when they encounter the multi-modal posterior distributions that are common for such models. SVGD provides a non-parametric alternative to variational inference which is substantially faster than MCMC. We prove that for GP models with Lipschitz gradients the SVGD algorithm monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior. Our method is demonstrated on benchmark problems in both regression and classification, a multimodal posterior, and an air quality example with 550,134 spatiotemporal observations, showing substantial performance improvements over MCMC and VI.

Citations

• Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
• Gaussian Processes for Big Data
• Elliptical slice sampling
• MCMC for Variationally Sparse Gaussian Processes
• A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• On the geometry of Stein variational gradient descent
• The equivalence between Stein variational gradient descent and black-box variational inference

Cited by

• A Probabilistic Assessment of the COVID-19 Lockdown on Air Quality in the UK

Related

• Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent
• Stein Variational Gradient Descent With Matrix-Valued Kernels
• Stein Variational Gradient Descent as Moment Matching
• Grassmann Stein Variational Gradient Descent
• Accelerated Stein Variational Gradient Flow