2021/07/19 by William Stephenson, Stephenson, William T., Zachary Frangella +5 · 2 citations
Engineering · Mathematics · Computer Science · #Sparse and Compressive Sensing Techniques #Statistical Methods and Inference #Face and Expression Recognition
paper · pdf · doi:10.48550/arxiv.2107.09194
Models like LASSO and ridge regression are extensively used in practice due\nto their interpretability, ease of use, and strong theoretical guarantees.\nCross-validation (CV) is widely used for hyperparameter tuning in these models,\nbut do practical optimization methods minimize the true out-of-sample loss? A\nrecent line of research promises to show that the optimum of the CV loss\nmatches the optimum of the out-of-sample loss (possibly after simple\ncorrections). It remains to show how tractable it is to minimize the CV loss.\nIn the present paper, we show that, in the case of ridge regression, the CV\nloss may fail to be quasiconvex and thus may have multiple local optima. We can\nguarantee that the CV loss is quasiconvex in at least one case: when the\nspectrum of the covariate matrix is nearly flat and the noise in the observed\nresponses is not too high. More generally, we show that quasiconvexity status\nis independent of many properties of the observed data (response norm,\ncovariate-matrix right singular vectors and singular-value scaling) and has a\ncomplex dependence on the few that remain. We empirically confirm our theory\nusing simulated experiments.\n