Bayesian Hyperparameter Optimization with BoTorch, GPyTorch and Ax

2019/12/11 by Daniel T. Chang, Chang, Daniel T. · 3 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Advanced Multi-Objective Optimization Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1912.05686

openalex publication_date 2019/12/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consuming and expensive model training / validation, is not a trivial challenge. Bayesian optimization is a powerful tool for the joint optimization of hyperparameters, efficiently trading off exploration and exploitation of the hyperparameter space. In this paper, we discuss Bayesian hyperparameter optimization, including hyperparameter optimization, Bayesian optimization, and Gaussian processes. We also review BoTorch, GPyTorch and Ax, the new open-source frameworks that we use for Bayesian optimization, Gaussian process inference and adaptive experimentation, respectively. For experimentation, we apply Bayesian hyperparameter optimization, for optimizing group weights, to weighted group pooling, which couples unsupervised tiered graph autoencoders learning and supervised graph prediction learning for molecular graphs. We find that Ax, BoTorch and GPyTorch together provide a simple-to-use but powerful framework for Bayesian hyperparameter optimization, using Ax's high-level API that constructs and runs a full optimization loop and returns the best hyperparameter configuration.

Citations

• Practical Bayesian Optimization of Machine Learning Algorithms
• Taking the Human Out of the Loop: A Review of Bayesian Optimization
• Bayesian Optimization for Machine Learning : A Practical Guidebook
• Tiered Latent Representations and Latent Spaces for Molecular Graphs

Cited by

• Automating Data-Driven Modeling and Analysis for Engineering Applications using Large Language Model Agents
• Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models
• Geometric Graph Representations and Geometric Graph Convolutions for\n Deep Learning on Three-Dimensional (3D) Graphs

Related

• Bayesian Hyperparameter Optimization for Ensemble Learning
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Scalable Hyperparameter Optimization with Lazy Gaussian Processes
• Pre-training helps Bayesian optimization too
• Supervising the Multi-Fidelity Race of Hyperparameter Configurations