2015/11/20 by Jelena Luketina, Mathias Berglund, Luketina, Jelena +5 · 18 citations
Computer Science · #Advanced Neural Network Applications #Artificial intelligence #Artificial neural network #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Hyperparameter #Hyperparameter optimization #MNIST database #Machine Learning (cs.LG) #Machine Learning and Data Classification #Machine learning #Model selection #Regularization (linguistics) #Scalability #Support vector machine #cs.LG
paper · pdf · doi:10.48550/arxiv.1511.06727
published in arXiv (Cornell University) (Cornell University) · 9 pages, 7 figures. Accepted at ICML 2016
openalex publication_date 2015/11/20 · arxiv created 2016/06/17 · arxiv updated 2016/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Hyperparameter selection generally relies on running multiple full training trials, with selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during training of the model. Hyperparameters are adjusted so as to make the model parameter gradients, and hence updates, more advantageous for the validation cost. We explore the approach for tuning regularization hyperparameters and find that in experiments on MNIST, SVHN and CIFAR-10, the resulting regularization levels are within the optimal regions. The additional computational cost depends on how frequently the hyperparameters are trained, but the tested scheme adds only 30% computational overhead regardless of the model size. Since the method is significantly less computationally demanding compared to similar gradient-based approaches to hyperparameter optimization, and consistently finds good hyperparameter values, it can be a useful tool for training neural network models.