GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

2018/09/28 by Jacob R. Gardner, Gardner, Jacob R., Geoff Pleiss +7 · 172 citations
Computer Science · Engineering · Mathematics · #Control Systems and Identification #Gaussian Processes and Bayesian Inference #Neural Networks and Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1809.11165

NeurIPS 2018. Most recent version includes additional details on preconditioned BBMM

arxiv created 2021/06/29 · arxiv updated 2021/07/01

Abstract

Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from O(n³) to O(n²). Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.

Citations

• Auto-Encoding Variational Bayes

Cited by

• A Novel Geometry-Aware GPR-Based Energy-Efficient and Low-Overhead Channel Estimation Scheme
• Variational Quantum Conditional Boltzmann Machines for Time-Series Forecasting: Architectures, Symmetric Hyperparameter Evaluation, and a Nonlinear Benchmark
• Amortized Posteriors for Estimation of Material Constitutive Parameters from Multimodal Measurements on Small Punch Tests
• Identification of periodicities with arbitrary shapes in AGN light curves
• Flow matching Operators for Residual-Augmented Probabilistic Learning of Partial Differential Equations
• A Stochastic Approach to Terrain Maps for Safe Lunar Landing
• Latent-Autoregressive GP-VAE Language Model
• All Emulators are Wrong, Many are Useful, and Some are More Useful Than Others: A Reproducible Comparison of Computer Model Surrogates
• Control-Oriented System Identification: Classical, Learning, and Physics-Informed Approaches
• High-Resolution Retrieval of Atmospheric Boundary Layers with Nonstationary Gaussian Processes
• Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores
• Multi-fidelity Bayesian Optimization Framework for CFD-Based Non-Premixed Burner Design
• Neural NMPC through Signed Distance Field Encoding for Collision Avoidance
• Operator Learning at Machine Precision
• Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
• Bayesian Calibration of Engine-out NOx Models for Engine-to-Engine Transferability
• Robust Bayesian Optimisation with Unbounded Corruptions
• On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty
• Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees
• SarcGraph for High-Throughput Regional Analysis of Sarcomere Organization and Contractile Function in 2D Cardiac Muscle Bundles
• Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification
• Amortized Multi-Objective Optimization Across Tasks with Generative Solution Modeling
• Consecutive Preferential Bayesian Optimization
• Uncertainty quantification and parameter optimization of plasma etching process using heteroscedastic Gaussian process
• Robust Variational Gaussian Process Regression for Count Data with the Trimmed Marginal Likelihood
• MVeLMA: Multimodal Vegetation Loss Modeling Architecture for Predicting Post-fire Vegetation Loss
• Uncertainty-Aware Diagnostics for Physics-Informed Machine Learning
• Spatio-Temporal Variational Gaussian Processes
• Distribution Regression for Sequential Data
• High-Dimensional Gaussian Process Inference with Derivatives
• Bayesian Layers: A Module for Neural Network Uncertainty
• Uncertainty-Aware Time-to-Event Prediction using Deep Kernel Accelerated Failure Time Models
• When Gaussian Process Meets Big Data: A Review of Scalable GPs
• ShaplEIG: Bayesian Experimental Design for Shapley Value Estimation
• Deep Bayesian Estimation for Dynamic Treatment Regimes with a Long Follow-up Time
• Fast Bayesian Multilevel Quasi-Monte Carlo
• Preconditioned Truncated Single-Sample Estimators for Scalable Stochastic Optimization
• Synergizing chemical and AI communities for advancing laboratories of the future
• Bayesian Optimization Algorithms for Accelerator Physics
• Informed Initialization for Bayesian Optimization and Active Learning
• Deep Jump Gaussian Processes for Surrogate Modeling of High-Dimensional Piecewise Continuous Functions
• MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions
• No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processes
• Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
• Is Multilingual LLM Watermarking Truly Multilingual? Scaling Robustness to 100+ Languages via Back-Translation
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Infinite Neural Operators: Gaussian processes on functions
• PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python
• Bayesian inference of the magnetic component of quark-gluon plasma
• nuGPR: GPU-Accelerated Gaussian Process Regression with Iterative Algorithms and Low-Rank Approximations
• A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels
• Multi-level informed optimization via decomposed Kriging for large design problems under uncertainty
• Multi-fidelity Batch Active Learning for Gaussian Process Classifiers
• A Guide to Bayesian Optimization in Bioprocess Engineering
• Proteome-wide model for human disease genetics
• NeST-BO: Fast Local Bayesian Optimization via Newton-Step Targeting of Gradient and Hessian Information
• Learning Passive Continuous-Time Dynamics with Multistep Port-Hamiltonian Gaussian Processes
• A Physics-Guided Probabilistic Surrogate Modeling Framework for Digital Twins of Underwater Radiated Noise
• Is Sequence Information All You Need for Bayesian Optimization of Antibodies?
• Meta-Learning Theory-Informed Inductive Biases using Deep Kernel Gaussian Processes
• BOSfM: A View Planning Framework for Optimal 3D Reconstruction of Agricultural Scenes
• Deconditional Downscaling with Gaussian Processes
• A Genetic Algorithm for Navigating Synthesizable Molecular Spaces
• Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling
• Bayesian Probabilistic Numerical Integration with Tree-Based Models
• Safe and Near-Optimal Control with Online Dynamics Learning
• Model Predictive Control with Reference Learning for Soft Robotic Intracranial Pressure Waveform Modulation
• Scaling Gaussian Processes with Derivative Information Using Variational Inference
• High-Dimensional Bayesian Optimization via Nested Riemannian Manifolds
• Meta-Learning Mean Functions for Gaussian Processes
• Exact Gaussian Processes on a Million Data Points
• Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures
• Graph Random Features for Scalable Gaussian Processes
• Scalable Constrained Bayesian Optimization
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• LFaB: Low fidelity as Bias for Active Learning in the chemical configuration space
• Generative AI Against Poaching: Latent Composite Flow Matching for Wildlife Conservation
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• DRF: A Framework for High-Accuracy Autonomous Driving Vehicle Modeling
• Large Wing Model
• Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next
• Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features
• Holistic Bioprocess Development Across Scales Using Multi-Fidelity Batch Bayesian Optimization
• Tighter Bounds on the Log Marginal Likelihood of Gaussian Process Regression Using Conjugate Gradients
• Enhancing Corrosion Resistance of Aluminum Alloys Through AI and ML Modeling
• Matrix Factorization-Based Solar Spectral Irradiance Missing Data Imputation with Uncertainty Quantification
• Uncertainty-aware Accurate Elevation Modeling for Off-road Navigation via Neural Processes
• Scalable neural network-based blackbox optimization
• Fast Gaussian process inference by exact Matérn kernel decomposition
• Stability-Constrained AC Optimal Power Flow--A Gaussian Process-Based Approach
• Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
• Novel Pivoted Cholesky Decompositions for Efficient Gaussian Process Inference
• Streamlined Empirical Bayes Fitting of Linear Mixed Models in Mobile Health
• Deep Factors with Gaussian Processes for Forecasting
• Gaussian Process Uniform Error Bounds with Unknown Hyperparameters for Safety-Critical Applications
• Adaptive Parameter Optimization in Gaussian Processes: A Comprehensive Study of Uncertainty Quantification and Dimensional Scaling
• Kernel-Matrix Determinant Estimates from stopped Cholesky Decomposition
• Barely Biased Learning for Gaussian Process Regression
• Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization
• From Sorting Algorithms to Scalable Kernels: Bayesian Optimization in High-Dimensional Permutation Spaces
• CorrAttack: Black-box Adversarial Attack with Structured Search
• Improving predictions of Bayesian neural nets via local linearization
• A Framework for Nonstationary Gaussian Processes with Neural Network Parameters
• Bayesian Variable Selection for Single Index Logistic Model
• Path Imputation Strategies for Signature Models of Irregular Time Series
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Neural Likelihoods for Multi-Output Gaussian Processes
• Multiobjective Aerodynamic Design Optimization of the NASA Common Research Model
• Gaussian Process Methods for Very Large Astrometric Data Sets
• SGPMIL: Sparse Gaussian Process Multiple Instance Learning
• Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes
• Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces
• A robust and adaptive MPC formulation for Gaussian process models
• Kernel Conditional Density Operators
• EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception
• Training Flexible Models of Genetic Variant Effects from Functional Annotations using Accelerated Linear Algebra
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Bayesian Optimization over Bounded Domains with the Beta Product Kernel
• Taking the GP Out of the Loop
• Spectral Estimation with Free Decompression
• Planning from Images with Deep Latent Gaussian Process Dynamics
• Scalable Gaussian Processes with Latent Kronecker Structure
• Machine learning for in-situ composition mapping in a self-driving magnetron sputtering system
• Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
• Bayesian Meta-Learning Through Variational Gaussian Processes
• Bayesian Optimization with High-Dimensional Outputs
• Balanced Depth Completion between Dense Depth Inference and Sparse Range Measurements via KISS-GP
• Bayesian Optimization from Human Feedback: Near-Optimal Regret Bounds
• TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
• Rectangular Flows for Manifold Learning
• Variability in the supermassive black hole binary candidate SDSS J2320+0024: No evidence of periodic modulation
• A Multi-output Gaussian Process Regression with Negative Transfer Mitigation for Generating Boundary Test Scenarios of Multi-UAV Systems
• STACI: Spatio-Temporal Aleatoric Conformal Inference
• Robust and Computation-Aware Gaussian Processes
• Scalable Deep Basis Kernel Gaussian Processes
• Joker: Joint Optimization Framework for Lightweight Kernel Machines
• Neighbour-Driven Gaussian Process Variational Autoencoders for Scalable Structured Latent Modelling
• Gaussian Processes in Power Systems: Techniques, Applications, and Future Works
• Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes
• Turbocharging Gaussian Process Inference with Approximate Sketch-and-Project
• Bayesian Optimization of Pythia8 Tunes
• Graph and Simplicial Complex Prediction Gaussian Process via the Hodgelet Representations
• NeuralSurv: Deep Survival Analysis with Bayesian Uncertainty Quantification
• Estimating Velocity Vector Fields of Atmospheric Winds using Transport Gaussian Processes
• Bidirectional Information Flow (BIF) -- A Sample Efficient Hierarchical Gaussian Process for Bayesian Optimization
• Fast and Accurate Gaussian Kernel Ridge Regression Using Matrix Decompositions for Preconditioning
• Agentic Bayesian Optimization through Surrogate-Augmented Autoresearch
• Towards scalable surrogate models based on Neural Fields for large scale aerodynamic simulations
• Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects
• Scaling Gaussian Process Regression with Full Derivative Observations
• Finite-Sample-Based Reachability for Safe Control with Gaussian Process Dynamics
• Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models
• A new gradient-free active subspace estimation method with application to rare event probability estimation
• Collaborative Multidisciplinary Design Optimization with Neural Networks
• Shape Representation using Gaussian Process mixture models
• Fast Multitask Gaussian Process Regression
• Learning and Interpreting Multi-Multi-Instance Learning Networks
• Risk factor identification for incident heart failure using neural network distillation and variable selection
• GPRat: Gaussian Process Regression with Asynchronous Tasks
• Trajectory-Oriented Optimization Via Adaptive Thompson Sampling And Grid Refinement: A Tutorial With The ADAPTIVE_TS Package
• Joint Model and Data Sparsification via the Marginal Likelihood
• On learning functions over biological sequence space: relating Gaussian process priors, regularization, and gauge fixing
• Pragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active Inference
• Hierarchical Recursive Precision for Accelerating Symmetric Linear Solves on MXUs
• Rapidly Personalizing Mobile Health Treatment Policies with Limited Data
• Confidence-Aware Learning Optimal Terminal Guidance via Gaussian Process Regression
• Multi-Sparse Gaussian Process: Learning based Semi-Parametric Control
• CPT-based two-dimensional soil stratification using non-stationary mixture of Gaussian processes
• Incorporating structured assumptions with probabilistic graphical models in fMRI data analysis
• The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopy
• Large language models as uncertainty-calibrated optimizers for experimental discovery
• Multiobjective Optimization under Uncertainties using Conditional Pareto Fronts
• Optimal Krylov On Average

Related

• Exact Gaussian Processes on a Million Data Points
• Scalable Gaussian Processes: Advances in Iterative Methods and Pathwise Conditioning
• nuGPR: GPU-Accelerated Gaussian Process Regression with Iterative Algorithms and Low-Rank Approximations
• Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference
• GPU-Accelerated Modified Bessel Function of the Second Kind for Gaussian Processes