Toward Optimal Run Racing: Application to Deep Learning Calibration

2017/06/10 by Olivier Bousquet, Sylvain Gelly, Bousquet, Olivier +11
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1706.03199

openalex publication_date 2017/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper aims at one-shot learning of deep neural nets, where a highly parallel setting is considered to address the algorithm calibration problem - selecting the best neural architecture and learning hyper-parameter values depending on the dataset at hand. The notoriously expensive calibration problem is optimally reduced by detecting and early stopping non-optimal runs. The theoretical contribution regards the optimality guarantees within the multiple hypothesis testing framework. Experimentations on the Cifar10, PTB and Wiki benchmarks demonstrate the relevance of the approach with a principled and consistent improvement on the state of the art with no extra hyper-parameter.

Citations

• Freeze-Thaw Bayesian Optimization
• Practical Bayesian Optimization of Machine Learning Algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual\n Recognition
• ImageNet classification with deep convolutional neural networks
• Bayesian Hyperparameter Optimization for Ensemble Learning

Related

• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
• Open Problems in Applied Deep Learning
• Calibrating a Deep Neural Network with Its Predecessors
• The observer-assisted method for adjusting hyper-parameters in deep learning algorithms
• Calibration-then-Calculation: A Variance Reduced Metric Framework in Deep Click-Through Rate Prediction Models