Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT

2021/05/30 by Thomas Bartz–Beielstein, Thomas Bartz-Beielstein, Bartz-Beielstein, Thomas +6
Computer Science · #68T07 #A.1 #B.8.0 #FOS: Computer and information sciences #G.1.6 #G.4 #I.2.8 #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification #Software Testing and Debugging Techniques #acm:68T07 #cs.LG #msc:68T07

paper · pdf · doi:10.48550/arxiv.2105.14625

version 3

openalex publication_date 2021/05/30 · arxiv created 2021/07/05 · arxiv updated 2021/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A surrogate model based hyperparameter tuning approach for deep learning is presented. This article demonstrates how the architecture-level parameters (hyperparameters) of deep learning models that were implemented in Keras/tensorflow can be optimized. The implementation of the tuning procedure is 100% accessible from R, the software environment for statistical computing. With a few lines of code, existing R packages (tfruns and SPOT) can be combined to perform hyperparameter tuning. An elementary hyperparameter tuning task (neural network and the MNIST data) is used to exemplify this approach

Citations

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• In a Nutshell -- The Sequential Parameter Optimization Toolbox

Related

• Guided Hyperparameter Tuning Through Visualization and Inference
• On tuning deep learning models: a data mining perspective
• PyTorch Hyperparameter Tuning - A Tutorial for spotPython
• Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
• Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training