Learning Curves for Gaussian Process Regression: Approximations and Bounds

2001/05/01 by Peter Sollich, Anason Halees · 1 citation
Computer Science · Mathematics · Physics and Astronomy · #Gaussian Processes and Bayesian Inference #Statistical Methods and Inference #Stochastic Gradient Optimization Techniques #cond-mat.dis-nn #cond-mat.stat-mech

paper · pdf · doi:10.1162/089976602753712990

published as Neural Computation, 14:1393-1428, 2002. · 25 pages, 10 figures

arxiv created 2001/05/01 · openalex publication_date 2002/06/01 · arxiv updated 2009/11/30 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

We consider the problem of calculating learning curves (i.e., average generalization performance) of gaussian processes used for regression. On the basis of a simple expression for the generalization error, in terms of the eigenvalue decomposition of the covariance function, we derive a number of approximation schemes. We identify where these become exact and compare with existing bounds on learning curves; the new approximations, which can be used for any input space dimension, generally get substantially closer to the truth. We also study possible improvements to our approximations. Finally, we use a simple exactly solvable learning scenario to show that there are limits of principle on the quality of approximations and bounds expressible solely in terms of the eigenvalue spectrum of the covariance function.

Citations

• A Practical Bayesian Framework for Backpropagation Networks

Cited by

• Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization

Related

• Regularity dependence of the rate of convergence of the learning curve for Gaussian process regression
• Learning curves for Gaussian process regression with power-law priors and targets
• Gaussian Process Regression with Mismatched Models
• Learning curves for multi-task Gaussian process regression
• Hilbert space methods for reduced-rank Gaussian process regression