Optimizing Approximate Leave-one-out Cross-validation to Tune\n Hyperparameters

2020/11/19 by Ryan Burn, Burn, Ryan
Computer Science · Mathematics · #Machine Learning and Data Classification #Statistical Methods and Inference #Gaussian Processes and Bayesian Inference

paper · pdf · doi:10.48550/arxiv.2011.10218

Abstract

For a large class of regularized models, leave-one-out cross-validation can\nbe efficiently estimated with an approximate leave-one-out formula (ALO). We\nconsider the problem of adjusting hyperparameters so as to optimize ALO. We\nderive efficient formulas to compute the gradient and hessian of ALO and show\nhow to apply a second-order optimizer to find hyperparameters. We demonstrate\nthe usefulness of the proposed approach by finding hyperparameters for\nregularized logistic regression and ridge regression on various real-world data\nsets.\n

Citations

• The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction

Related

• Evaluation and Optimization of Leave-one-out Cross-validation for the Lasso
• Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference
• Approximate Leave-one-out Cross Validation for Regression with ℓ₁ Regularizers (extended version)
• Bayes beats Cross Validation: Efficient and Accurate Ridge Regression via Expectation Maximization
• Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions