2016/06/01 by Ning Xu, Jian Hong, Xu, Ning +3
Economics, Econometrics and Finance · Mathematics · Social Sciences · #Computation (stat.CO) #Economic Growth and Productivity #Economic Policies and Impacts #FOS: Computer and information sciences #FOS: Economics and business #General Economics (econ.GN) #Insurance, Mortality, Demography, Risk Management #Machine Learning (stat.ML) #Spatial and Panel Data Analysis #Statistical Methods and Inference
paper · pdf · doi:10.48550/arxiv.1606.00142
openalex publication_date 2016/06/01 · openalex created_date 2019/07/30 · openalex updated_date 2026/07/28
Model selection is difficult to analyse yet theoretically and empirically\nimportant, especially for high-dimensional data analysis. Recently the least\nabsolute shrinkage and selection operator (Lasso) has been applied in the\nstatistical and econometric literature. Consis- tency of Lasso has been\nestablished under various conditions, some of which are difficult to verify in\npractice. In this paper, we study model selection from the perspective of\ngeneralization ability, under the framework of structural risk minimization\n(SRM) and Vapnik-Chervonenkis (VC) theory. The approach emphasizes the balance\nbetween the in-sample and out-of-sample fit, which can be achieved by using\ncross-validation to select a penalty on model complexity. We show that an exact\nrelationship exists between the generalization ability of a model and model\nselection consistency. By implementing SRM and the VC inequality, we show that\nLasso is L2-consistent for model selection under assumptions similar to those\nimposed on OLS. Furthermore, we derive a probabilistic bound for the distance\nbetween the penalized extremum estimator and the extremum estimator without\npenalty, which is dominated by overfitting. We also propose a new measurement\nof overfitting, GR2, based on generalization ability, that converges to zero if\nmodel selection is consistent. Using simulations, we demonstrate that the\nproposed CV-Lasso algorithm performs well in terms of model selection and\noverfitting control.\n