2013/11/12 by Shachar Kaufman, Saharon Rosset, Kaufman, Shachar +1 · 1 citation
Mathematics · Decision Sciences · Computer Science · #Statistical Methods and Inference #Grey System Theory Applications #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1311.2791
Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected behavior does not hold in general. In fact, counter examples are given that show regularization can increase the degrees of freedom in simple situations, including lasso and ridge regression, which are the most common regularization approaches in use. In such situations, the regularization increases both training error and degrees of freedom, and is thus inherently without merit. On the other hand, two important regularization scenarios are described where the expected reduction in degrees of freedom is indeed guaranteed: (a) all symmetric linear smoothers, and (b) linear regression versus convex constrained linear regression (as in the constrained variant of ridge regression and lasso).