2020/10/19 by Eshaan Nichani, Nichani, Eshaan, Adityanarayanan Radhakrishnan +3
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification
paper · pdf · doi:10.48550/arxiv.2010.09610
openalex publication_date 2020/10/19 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Recent works have demonstrated that increasing model capacity through width\nin over-parameterized neural networks leads to a decrease in test risk. For\nneural networks, however, model capacity can also be increased through depth,\nyet understanding the impact of increasing depth on test risk remains an open\nquestion. In this work, we demonstrate that the test risk of over-parameterized\nconvolutional networks is a U-shaped curve (i.e. monotonically decreasing, then\nincreasing) with increasing depth. We first provide empirical evidence for this\nphenomenon via image classification experiments using both ResNets and the\nconvolutional neural tangent kernel (CNTK). We then present a novel linear\nregression framework for characterizing the impact of depth on test risk, and\nshow that increasing depth leads to a U-shaped test risk for the linear CNTK.\nIn particular, we prove that the linear CNTK corresponds to a depth-dependent\nlinear transformation on the original space and characterize properties of this\ntransformation. We then analyze over-parameterized linear regression under\narbitrary linear transformations and, in simplified settings, provably identify\nthe depths which minimize each of the bias and variance terms of the test risk.\n