Towards an Understanding of Benign Overfitting in Neural Networks

2021/06/06 by Li Zhu, Zhi‐Hua Zhou, Li, Zhu +3
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2106.03212

openalex publication_date 2021/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss; yet surprisingly, they possess near-optimal prediction performance, contradicting classical learning theory. We examine how these benign overfitting phenomena occur in a two-layer neural network setting where sample covariates are corrupted with noise. We address the high dimensional regime, where the data dimension d grows with the number n of data points. Our analysis combines an upper bound on the bias with matching upper and lower bounds on the variance of the interpolator (an estimator that interpolates the data). These results indicate that the excess learning risk of the interpolator decays under mild conditions. We further show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate, which to our knowledge is the first generalization result for such networks. Finally, our theory predicts that the excess learning risk starts to increase once the number of parameters s grows beyond O(n²), matching recent empirical findings.

Citations

• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Optimal Regularization Can Mitigate Double Descent
• On the Optimal Weighted ℓ₂ Regularization in Overparameterized Linear Regression
• Risk Bounds for Over-parameterized Maximum Margin Classification on\n Sub-Gaussian Mixtures

Related

• Noisy Interpolation Learning with Shallow Univariate ReLU Networks
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks
• A Classical View on Benign Overfitting: The Role of Sample Size
• Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
• Benign Overfitting for Two-layer ReLU Convolutional Neural Networks