2022/02/23 by Yue Xing, Xing, Yue, Qifan Song +3
Computer Science · #Machine Learning and Data Classification #Advanced Neural Network Applications #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2202.11817
In some studies \citep[e.g.,][]zhang2016understanding of deep learning, it is observed that over-parametrized deep neural networks achieve a small testing error even when the training error is almost zero. Despite numerous works towards understanding this so-called "double descent" phenomenon \citep[e.g.,][]belkin2018reconciling,belkin2019two, in this paper, we turn into another way to enforce zero training error (without over-parametrization) through a data interpolation mechanism. Specifically, we consider a class of interpolated weighting schemes in the nearest neighbors (NN) algorithms. By carefully characterizing the multiplicative constant in the statistical risk, we reveal a U-shaped performance curve for the level of data interpolation in both classification and regression setups. This sharpens the existing result \citepbelkin2018does that zero training error does not necessarily jeopardize predictive performances and claims a counter-intuitive result that a mild degree of data interpolation actually \em strictly improve the prediction performance and statistical stability over those of the (un-interpolated) k-NN algorithm. In the end, the universality of our results, such as change of distance measure and corrupted testing data, will also be discussed.