2019/03/31 by Mikhail Belkin, Daniel Hsu, Ji Xu
Computer Science · Mathematics · #cs.LG #stat.ML
paper · pdf · doi:10.1137/20m1336072
published as SIAM Journal on Mathematics of Data Science, 2(4):1167-1180, 2020
arxiv created 2020/10/10 · arxiv updated 2020/12/22
The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features p is close to the sample size n, but also that the risk decreases towards its minimum as p increases beyond n. This behavior is contrasted with that of "prescient" models that select features in an a priori optimal order.