Over-parametrized deep neural networks do not generalize well

2019/12/09 by Kohler, Michael, Krzyzak, Adam
#FOS: Mathematics #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.1912.03925

Abstract

Recently it was shown in several papers that backpropagation is able to find the global minimum of the empirical risk on the training data using over-parametrized deep neural networks. In this paper a similar result is shown for deep neural networks with the sigmoidal squasher activation function in a regression setting, and a lower bound is presented which proves that these networks do not generalize well on a new data in the sense that they do not achieve the optimal minimax rate of convergence for estimation of smooth regression functions.

Related

• Nearly Minimal Over-Parametrization of Shallow Neural Networks
• Analysis of the expected L₂ error of an over-parametrized deep neural network estimate learned by gradient descent without regularization
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• On the rate of convergence of an over-parametrized deep neural network regression estimate learned by gradient descent
• Minimum weight norm models do not always generalize well for over-parameterized problems