Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models

2021/01/06 by Jeroen van Hoof, Joaquin Vanschoren, van Hoof, Jeroen +1 · 3 citations
Computer Science · #Advanced Multi-Objective Optimization Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2101.02289

openalex publication_date 2021/01/06 · openalex created_date 2021/01/18 · openalex updated_date 2026/07/28

Abstract

Bayesian Optimization is a popular tool for tuning algorithms in automatic machine learning (AutoML) systems. Current state-of-the-art methods leverage Random Forests or Gaussian processes to build a surrogate model that predicts algorithm performance given a certain set of hyperparameter settings. In this paper, we propose a new surrogate model based on gradient boosting, where we use quantile regression to provide optimistic estimates of the performance of an unobserved hyperparameter setting, and combine this with a distance metric between unobserved and observed hyperparameter settings to help regulate exploration. We demonstrate empirically that the new method is able to outperform some state-of-the art techniques across a reasonable sized set of classification problems.

Citations

• Scalable Bayesian Optimization Using Deep Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• CatBoost: gradient boosting with categorical features support
• Single-Model Uncertainties for Deep Learning

Cited by

• Active Learning Driven Materials Discovery for Low Thermal Conductivity Rare-Earth Pyrochlore for Thermal Barrier Coatings
• Enhancing Performance and Calibration in Quantile Hyperparameter Optimization
• LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms

Related

• To tune or not to tune? An Approach for Recommending Important Hyperparameters
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• Better call Surrogates: A hybrid Evolutionary Algorithm for Hyperparameter optimization
• Hyperparameter Optimization in Machine Learning
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets