Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

2021/04/20 by Ryan Turner, David Eriksson, Turner, Ryan +11 · 21 citations
Computer Science · Mathematics · #Advanced Multi-Objective Optimization Algorithms #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2104.10201

openalex publication_date 2021/04/20 · openalex created_date 2021/04/26 · arxiv created 2021/08/31 · arxiv updated 2021/09/01 · openalex updated_date 2026/07/28

Abstract

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

Citations

• On the limited memory BFGS method for large scale optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• Predictive Entropy Search for Bayesian Optimization with Unknown Constraints
• A Simplex Method for Function Minimization
• Batch Bayesian Optimization via Local Penalization
• GPflowOpt: A Bayesian Optimization Library using TensorFlow
• GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms

Cited by

• PDx -- Adaptive Credit Risk Forecasting Model in Digital Lending using Machine Learning Operations
• DeepCAVE: A Visualization and Analysis Tool for Automated Machine Learning
• Dynamic Priors in Bayesian Optimization for Hyperparameter Optimization
• GPTOpt: Towards Efficient LLM-Based Black-Box Optimization
• OptunaHub: A Platform for Black-Box Optimization
• ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization
• Stochastic Optimization with Random Search
• Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models
• Holistic Bioprocess Development Across Scales Using Multi-Fidelity Batch Bayesian Optimization
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Bayesian Optimization with Inexact Acquisition: Is Random Grid Search Sufficient?
• Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings
• carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks
• LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms
• Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
• Improving LLM-based Global Optimization with Search Space Partitioning
• A Generative Neural Annealer for Black-Box Combinatorial Optimization
• Language Model Crossover: Variation through Few-Shot Prompting
• ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization
• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization
• Bayesian Optimization in Linear Time

Related

• Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization
• Towards Assessing the Impact of Bayesian Optimization's Own\n Hyperparameters
• Towards Automatic Bayesian Optimization: A first step involving acquisition functions
• Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking
• Adaptive Local Bayesian Optimization Over Multiple Discrete Variables