Hilbert space methods for reduced-rank Gaussian process regression

2014/01/31 by Arno Solin, Simo Särkkä · 1 voice · 134 citations
Computer Science · Mathematics · #Advanced Multi-Objective Optimization Algorithms #Covariance #Covariance function #Covariance operator #Eigenfunction #Gaussian Processes and Bayesian Inference #Gaussian process #Hilbert space #Matérn covariance function #Rational quadratic covariance function #Reproducing kernel Hilbert space #Stochastic Gradient Optimization Techniques #stat.ML

paper · pdf · doi:10.1007/s11222-019-09886-w

published in Statistics and Computing 30(2), 419-446 (Springer Science+Business Media)

openalex created_date 2016/06/24 · arxiv created 2019/08/01 · openalex publication_date 2019/08/05 · arxiv updated 2020/06/26 · openalex updated_date 2026/08/06

Abstract

This paper proposes a novel scheme for reduced-rank Gaussian process regression. The method is based on an approximate series expansion of the covariance function in terms of an eigenfunction expansion of the Laplace operator in a compact subset of \(\mathbb R^d\) . On this approximate eigenbasis, the eigenvalues of the covariance function can be expressed as simple functions of the spectral density of the Gaussian process, which allows the GP inference to be solved under a computational cost scaling as \(\mathcal O(nm²)\) (initial) and \(\mathcal O(m³)\) (hyperparameter learning) with m basis functions and n data points. Furthermore, the basis functions are independent of the parameters of the covariance function, which allows for very fast hyperparameter learning. The approach also allows for rigorous error analysis with Hilbert space theory, and we show that the approximation becomes exact when the size of the compact subset and the number of eigenfunctions go to infinity. We also show that the convergence rate of the truncation error is independent of the input dimensionality provided that the differentiability order of the covariance function increases appropriately, and for the squared exponential covariance function it is always bounded by \(∼ 1/m\) regardless of the input dimensionality. The expansion generalizes to Hilbert spaces with an inner product which is defined as an integral over a specified input density. The method is compared to previously proposed methods theoretically and through empirical tests with simulated and real data.

Cited by

• Differentiable latent structure discovery for interpretable forecasting in clinical time series
• Conditioning Gaussian Processes on Almost Anything
• Laplace Approximation For Tensor Train Kernel Machines In System Identification
• Discovering Spatial Patterns of Readmission Risk Using a Bayesian Competing Risks Model with Spatially Varying Coefficients
• Gaussian process priors with Markov properties for effective reproduction number inference
• Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models
• Learning Dynamics from Input-Output Data with Hamiltonian Gaussian Processes
• Latent variable estimation with composite Hilbert space Gaussian processes
• Cross-national analyses require additional controls to account for the non-independence of nations
• A Bayesian Gaussian Process Dynamic Factor Model
• Bayesian Inference and Learning in Nonlinear Dynamical Systems: A Framework for Incorporating Explicit and Implicit Prior Knowledge
• Quantum-assisted Gaussian process regression using random Fourier features
• Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization
• Interpretable Bayesian Tensor Network Kernel Machines with Automatic Rank and Feature Selection
• Determination of Particle-Size Distributions from Light-Scattering Measurement Using Constrained Gaussian Process Regression
• Improved Regret Bounds for Gaussian Process Upper Confidence Bound in Bayesian Optimization
• Assessing Quantum Advantage for Gaussian Process Regression
• Efficient Deconvolution in Populational Inverse Problems
• Hilbert space methods for approximating multi-output latent variable Gaussian processes
• Data-driven inference of Hopf normal form representations from oscillatory time series
• Geometry-aware Active Learning of Spatiotemporal Dynamic Systems
• Constrained Gaussian Process Motion Planning via Stein Variational Newton Inference
• Gaussian process manifold interpolation for probabilistic atrial activation maps and uncertain conduction velocity. [europepmc]
• A Bayesian semi-parametric model for thermal proteome profiling. [europepmc]
• Spatiotemporal analysis of the first wave of COVID-19 hospitalisations in Birmingham, UK. [europepmc]
• Managing African Swine Fever: Assessing the Potential of Camera Traps in Monitoring Wild Boar Occupancy Trends in Infected and Non-infected Zones, Using Spatio-Temporal Statistical Models. [europepmc]
• Toward Integrated Large-Scale Environmental Monitoring Using WSN/UAV/Crowdsensing: A Review of Applications, Signal Processing, and Future Perspectives. [europepmc]
• An Extended Kalman Filter for Magnetic Field SLAM Using Gaussian Process Regression. [europepmc]
• Atrial conduction velocity mapping: clinical tools, algorithms and approaches for understanding the arrhythmogenic substrate. [europepmc]
• Calibrating cardiac electrophysiology models using latent Gaussian processes on atrial manifolds. [europepmc]
• Estimating fine age structure and time trends in human contact patterns from coarse contact data: The Bayesian rate consistency model. [europepmc]
• PyMC: a modern, and comprehensive probabilistic programming framework in Python. [europepmc]
• Assessing trends in population size of three unmarked species: A comparison of a multi-species N-mixture model and random encounter models. [europepmc]
• Bayesian workflow for time-varying transmission in stratified compartmental infectious disease transmission models. [europepmc]
• BOATMAP: Bayesian Optimization Active Targeting for Monomorphic Arrhythmia Pace-mapping. [europepmc]
• Low cost and real-time surveillance of enteric infection and diarrhoeal disease using rapid diagnostic tests in Cox's Bazar, Bangladesh. [europepmc]
• Addressing survey fatigue bias in longitudinal social contact studies to improve pandemic preparedness. [europepmc]
• Prevalence of malnutrition among hospital admissions to English NHS hospitals over two economically constrained decades: evidence from routine health data. [europepmc]
• A Bayesian model for repeated cross-sectional epidemic prevalence survey data. [europepmc]
• Estimating fine age structure and time trends in human contact patterns from coarse contact data: the Bayesian rate consistency model [europepmc]

Discussions

• A colleague today asked about Gaussian Processes on a sphere. Like spatial autocorrelation on the globe. I asked @avehtari.bsky.social. There are papers! This problem common enough to be worth having [bsky, 47 points, 6 comments]

Related

• Efficient reduced-rank methods for Gaussian processes with eigenfunction expansions
• Fast Approximate Multi-output Gaussian Processes
• Learning Integral Representations of Gaussian Processes
• Learning Curves for Gaussian Process Regression: Approximations and Bounds
• How Good are Low-Rank Approximations in Gaussian Process Regression?