Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

2020/07/06 by Hu, Tianyang, Wang, Wenjia, Lin, Cong +1 · 2 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2007.02486

Abstract

Overparametrized neural networks trained by gradient descent (GD) can provably overfit any training data. However, the generalization guarantee may not hold for noisy data. From a nonparametric perspective, this paper studies how well overparametrized neural networks can recover the true target function in the presence of random noises. We establish a lower bound on the L₂ estimation error with respect to the GD iterations, which is away from zero without a delicate scheme of early stopping. In turn, through a comprehensive analysis of ℓ₂-regularized GD trajectories, we prove that for overparametrized one-hidden-layer ReLU neural network with the ℓ₂ regularization: (1) the output is close to that of the kernel ridge regression with the corresponding neural tangent kernel; (2) minimax optimal rate of L₂ estimation error can be achieved. Numerical experiments confirm our theory and further demonstrate that the ℓ₂ regularization approach improves the training robustness and works for a wider range of neural networks.

Cited by

• Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention
• Sampling Complexity of TD and PPO in RKHS

Related

• Explicit Regularization in Overparametrized Models via Noise Injection
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Implicit Regularization in Over-parameterized Neural Networks
• Statistical Guarantees for Regularized Neural Networks
• Nonparametric Regression with Shallow Overparameterized Neural Networks\n Trained by GD with Early Stopping