Analysis of the expected L₂ error of an over-parametrized deep neural network estimate learned by gradient descent without regularization

2023/11/24 by Selina Drews, Drews, Selina, Michael Köhler +1 · 1 citation
Medicine · Computer Science · #Radiomics and Machine Learning in Medical Imaging #Stochastic Gradient Optimization Techniques #Medical Imaging Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2311.14609

Abstract

Recent results show that estimates defined by over-parametrized deep neural networks learned by applying gradient descent to a regularized empirical L₂ risk are universally consistent and achieve good rates of convergence. In this paper, we show that the regularization term is not necessary to obtain similar results. In the case of a suitably chosen initialization of the network, a suitable number of gradient descent steps, and a suitable step size we show that an estimate without a regularization term is universally consistent for bounded predictor variables. Additionally, we show that if the regression function is Hölder smooth with Hölder exponent 1/2 ≤ p ≤ 1, the L₂ error converges to zero with a convergence rate of approximately n^-1/(1+d). Furthermore, in case of an interaction model, where the regression function consists of a sum of Hölder smooth functions with d^* components, a rate of convergence is derived which does not depend on the input dimension d.

Cited by

• Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

Related

• Analysis of the rate of convergence of an over-parametrized deep neural network estimate learned by gradient descent
• Over-parametrized deep neural networks do not generalize well
• On the universal consistency of an over-parametrized deep neural network estimate learned by gradient descent
• Deep learning: a statistical viewpoint
• On the rate of convergence of an over-parametrized deep neural network regression estimate learned by gradient descent