Data-driven calibration of linear estimators with minimal penalties

2009/09/10 by Sylvain Arlot, Francis Bach, Arlot, Sylvain +1 · 2 citations
Engineering · Mathematics · #Advanced Statistical Methods and Models #Control Systems and Identification #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Methodology (stat.ME) #Statistical Methods and Inference #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.0909.1884

openalex publication_date 2009/09/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression, spline smoothing or locally weighted regression, and the choice of a kernel in multiple kernel learning. We propose a new algorithm which first estimates consistently the variance of the noise, based upon the concept of minimal penalty, which was previously introduced in the context of model selection. Then, plugging our variance estimate in Mallows' C_L penalty is proved to lead to an algorithm satisfying an oracle inequality. Simulation experiments with kernel ridge regression and multiple kernel learning show that the proposed algorithm often improves significantly existing calibration procedures such as generalized cross-validation.

Citations

• The Use of the L-Curve in the Regularization of Discrete Ill-Posed Problems
• The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction
• Consistency of the group Lasso and multiple kernel learning
• Smoothing noisy data with spline functions

Cited by

• The cost-free nature of optimally tuning Tikhonov regularizers and other ordered smoothers
• Collaborative Prediction: To Join or To Disjoin Datasets

Related

• Variable Selection for Nonparametric Learning with Power Series Kernels
• Model selection by resampling penalization
• Penalized Projected Kernel Calibration for Computer Models
• On the Improved Rates of Convergence for Matérn-type Kernel Ridge Regression, with Application to Calibration of Computer Models
• Non-asymptotic model selection for linear non least-squares estimation in regression models and inverse problems