Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

2014/09/14 by Kevin Swersky, David Duvenaud, Swersky, Kevin +7 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #stat.ML

paper · pdf · doi:10.48550/arxiv.1409.4011

6 pages, 3 figures. Appeared in the NIPS 2013 workshop on Bayesian optimization

arxiv created 2014/09/14 · openalex publication_date 2014/09/14 · arxiv updated 2014/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Gradient-based learning applied to document recognition
• Slice sampling covariance hyperparameters of latent Gaussian models
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Fast Learning Algorithm for Deep Belief Nets
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• ImageNet classification with deep convolutional neural networks
• A Kernel for Hierarchical Parameter Spaces
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

Related

• Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport
• Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization
• BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters