On the universal consistency of an over-parametrized deep neural network estimate learned by gradient descent

2022/08/30 by Selina Drews, Drews, Selina, Michael Köhler +1 · 1 citation
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Mathematics #Numerical methods in inverse problems #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2208.14283

openalex publication_date 2022/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Estimation of a multivariate regression function from independent and identically distributed data is considered. An estimate is defined which fits a deep neural network consisting of a large number of fully connected neural networks, which are computed in parallel, via gradient descent to the data. The estimate is over-parametrized in the sense that the number of its parameters is much larger than the sample size. It is shown that in case of a suitable random initialization of the network, a suitable small stepsize of the gradient descent, and a number of gradient descent steps which is slightly larger than the reciprocal of the stepsize of the gradient descent, the estimate is universally consistent in the sense that its expected L2 error converges to zero for all distributions of the data where the response variable is square integrable.

Cited by

• Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

Related

• Analysis of the rate of convergence of an over-parametrized deep neural network estimate learned by gradient descent
• Analysis of the expected L₂ error of an over-parametrized deep neural network estimate learned by gradient descent without regularization
• On the rate of convergence of an over-parametrized deep neural network regression estimate learned by gradient descent
• Over-parametrized deep neural networks do not generalize well
• Deep learning: a statistical viewpoint