PFNs4BO: In-Context Learning for Bayesian Optimization

2023/05/27 by Samuel Müller, Müller, Samuel, Matthias Feurer +5 · 15 citations
Computer Science · Decision Sciences · #FOS: Computer and information sciences #Forecasting Techniques and Applications #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2305.17535

openalex publication_date 2023/05/27 · openalex created_date 2023/05/31 · openalex updated_date 2026/07/28

Abstract

In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) through in-context learning on any prior distribution that can be efficiently sampled from. We describe how this flexibility can be exploited for surrogate modeling in BO. We use PFNs to mimic a naive Gaussian process (GP), an advanced GP, and a Bayesian Neural Network (BNN). In addition, we show how to incorporate further information into the prior, such as allowing hints about the position of optima (user priors), ignoring irrelevant dimensions, and performing non-myopic BO by learning the acquisition function. The flexibility underlying these extensions opens up vast possibilities for using PFNs for BO. We demonstrate the usefulness of PFNs for BO in a large-scale evaluation on artificial GP samples and three different hyperparameter optimization testbeds: HPO-B, Bayesmark, and PD1. We publish code alongside trained models at github.com/automl/PFNs4BO.

Cited by

• Uncertainty-Aware Data-Efficient AI: An Information-Theoretic Perspective
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• MIST: Mutual Information Estimation Via Supervised Training
• Parametric Pareto Set Learning for Expensive Multi-Objective Optimization
• Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization
• None To Optima in Few Shots: Bayesian Optimization with MDP Priors
• GPTOpt: Towards Efficient LLM-Based Black-Box Optimization
• OptunaHub: A Platform for Black-Box Optimization
• ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization
• Informed Initialization for Bayesian Optimization and Active Learning
• Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning
• Efficient Autoregressive Inference for Transformer Probabilistic Models
• GUIDe: Generative and Uncertainty-Informed Inverse Design for On-Demand Nonlinear Functional Responses
• FoMEMO: Towards Foundation Models for Expensive Multi-objective Optimization
• In-Context Decision Making for Optimizing Complex AutoML Pipelines

Related

• Transformers Can Do Bayesian Inference
• In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization
• Pre-training helps Bayesian optimization too
• Position: The Future of Bayesian Prediction Is Prior-Fitted
• Towards Scalable Bayesian Optimization via Gradient-Informed Bayesian Neural Networks