Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

2024/10/11 by Jihao Andreas Lin, Sebastian Ament, Lin, Jihao Andreas +5 · 4 citations
Computer Science · Decision Sciences · #Gaussian Processes and Bayesian Inference #Time Series Analysis and Forecasting #Multidisciplinary Science and Engineering Research

paper · pdf · doi:10.48550/arxiv.2410.09239

Abstract

A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, naïve GPs require O(n³m³) time and O(n² m²) space for n hyper-parameter configurations and O(m) learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose latent Kronecker structure to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires O(n³ + m³) time and O(n² + m²) space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.

Cited by

• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Zero-Shot Performance Prediction for Probabilistic Scaling Laws
• Graph Random Features for Scalable Gaussian Processes
• Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes

Related

• Successive Halving with Learning Curve Prediction via Latent Kronecker Gaussian Processes
• Scalable Gaussian Processes with Latent Kronecker Structure
• AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models
• Thoughts on Massively Scalable Gaussian Processes
• Active Learning of Linear Embeddings for Gaussian Processes