Optimizing for Generalization in Machine Learning with Cross-Validation Gradients

2018/05/18 by Barratt, Shane, Sharma, Rishi
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1805.07072

Abstract

Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for many common machine learning algorithms, including logistic regression, elastic-net regression, and support vector machines. Leveraging this property of differentiability, we propose a cross-validation gradient method (CVGM) for hyperparameter optimization. Our method enables efficient optimization in high-dimensional hyperparameter spaces of the cross-validation risk, the best surrogate of the true generalization ability of our learning algorithm.

Related

• Toward Theoretical Guidance for Two Common Questions in Practical Cross-Validation based Hyperparameter Selection
• Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
• Muddling Labels for Regularization, a novel approach to generalization
• Approximate Cross-validation: Guarantees for Model Assessment and Selection
• Nested cross-validation when selecting classifiers is overzealous for\n most practical applications