Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference

2025/01/05 by Bellec, Pierre C · 1 citation
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2501.02624

Abstract

Approximate Leave-One-Out Cross-Validation (ALO-CV) is a method that has been proposed to estimate the generalization error of a regularized estimator in the high-dimensional regime where dimension and sample size are of the same order, the so called ``proportional regime''. A new analysis is developed to derive the consistency of ALO-CV for non-differentiable regularizer under Gaussian covariates and strong-convexity of the regularizer. Using a conditioning argument, the difference between the ALO-CV weights and their counterparts in mean-field inference is shown to be small. Combined with upper bounds between the mean-field inference estimate and the leave-one-out quantity, this provides a proof that ALO-CV approximates the leave-one-out quantity as well up to negligible error terms. Linear models with square loss, robust linear regression and single-index models are explicitly treated.

Cited by

• Fast Leave-One-Out Approximation from Fragment-Target Prevalence Vectors (molFTP) : From Dummy Masking to Key-LOO for Leakage-Free Feature Construction

Related

• Approximate Leave-one-out Cross Validation for Regression with ℓ₁ Regularizers (extended version)
• Optimizing Approximate Leave-one-out Cross-validation to Tune\n Hyperparameters
• Theoretical Analysis of Leave-one-out Cross Validation for Non-differentiable Penalties under High-dimensional Settings
• A scalable estimate of the extra-sample prediction error via approximate leave-one-out
• Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions