Computationally Efficient High-Dimensional Bayesian Optimization via Variable Selection

2021/09/20 by Yihang Shen, Carl Kingsford, Shen, Yihang +1 · 6 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Advanced Multi-Objective Optimization Algorithms #Algorithm #Artificial intelligence #Bayesian optimization #Bayesian probability #Black box #Computer science #Dimension (graph theory) #Embedding #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #High dimensional #Hyperparameter #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematical optimization #Mathematics #Regret #Variable (mathematics)

paper · pdf · doi:10.48550/arxiv.2109.09264

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Bayesian Optimization (BO) is a method for globally optimizing black-box functions. While BO has been successfully applied to many scenarios, developing effective BO algorithms that scale to functions with high-dimensional domains is still a challenge. Optimizing such functions by vanilla BO is extremely time-consuming. Alternative strategies for high-dimensional BO that are based on the idea of embedding the high-dimensional space to the one with low dimension are sensitive to the choice of the embedding dimension, which needs to be pre-specified. We develop a new computationally efficient high-dimensional BO method that exploits variable selection. Our method is able to automatically learn axis-aligned sub-spaces, i.e. spaces containing selected variables, without the demand of any pre-specified hyperparameters. We theoretically analyze the computational complexity of our algorithm and derive the regret bound. We empirically show the efficacy of our method on several synthetic and real problems.

Citations

• Automated Machine Learning on Big Data using Stochastic Algorithm Tuning
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Practical Bayesian Optimization of Machine Learning Algorithms
• Minimum Regret Search for Single- and Multi-Task Optimization
• Bayesian Optimization for Synthetic Gene Design
• Constrained Bayesian Optimization for Automatic Chemical Design
• A Tutorial on Bayesian Optimization
• Bayesian Optimization with Dimension Scheduling: Application to\n Biological Systems
• Batched High-dimensional Bayesian Optimization via Structural Kernel Learning
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces

Cited by

• Efficient Sliced Wasserstein Distance Computation via Adaptive Bayesian Optimization
• Enhancing Trust-Region Bayesian Optimization via Newton Methods
• Data-Driven Cellular Mobility Management via Bayesian Optimization and Reinforcement Learning
• Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
• BONSAI: Bayesian Optimization with Natural Simplicity and Interpretability
• Cellular Network Design for UAV Corridors via Data-driven High-dimensional Bayesian Optimization
• Leveraging Axis-Aligned Subspaces for High-Dimensional Bayesian Optimization with Group Testing

Related

• High Dimensional Bayesian Optimization using Lasso Variable Selection
• BOIDS: High-dimensional Bayesian Optimization via Incumbent-guided Direction Lines and Subspace Embeddings
• Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization
• An Adaptive Dimension Reduction Estimation Method for High-dimensional Bayesian Optimization
• High-dimensional Bayesian optimization using low-dimensional feature spaces