Towards Improved Learning in Gaussian Processes: The Best of Two Worlds

2022/11/11 by Li, Rui, John, ST, Solin, Arno
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2211.06260

Abstract

Gaussian process training decomposes into inference of the (approximate) posterior and learning of the hyperparameters. For non-Gaussian (non-conjugate) likelihoods, two common choices for approximate inference are Expectation Propagation (EP) and Variational Inference (VI), which have complementary strengths and weaknesses. While VI's lower bound to the marginal likelihood is a suitable objective for inferring the approximate posterior, it does not automatically imply it is a good learning objective for hyperparameter optimization. We design a hybrid training procedure where the inference leverages conjugate-computation VI and the learning uses an EP-like marginal likelihood approximation. We empirically demonstrate on binary classification that this provides a good learning objective and generalizes better.

Related

• Improving Hyperparameter Learning under Approximate Inference in Gaussian Process Models
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients
• Modulating Scalable Gaussian Processes for Expressive Statistical Learning
• New Bounds for Sparse Variational Gaussian Processes