Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent

2020/02/12 by David Holzmüller, Holzmüller, David, Ingo Steinwart +1
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2002.04861

openalex publication_date 2020/02/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We prove that two-layer (Leaky)ReLU networks initialized by e.g. the widely used method proposed by He et al. (2015) and trained using gradient descent on a least-squares loss are not universally consistent. Specifically, we describe a large class of one-dimensional data-generating distributions for which, with high probability, gradient descent only finds a bad local minimum of the optimization landscape, since it is unable to move the biases far away from their initialization at zero. It turns out that in these cases, the found network essentially performs linear regression even if the target function is non-linear. We further provide numerical evidence that this happens in practical situations, for some multi-dimensional distributions and that stochastic gradient descent exhibits similar behavior. We also provide empirical results on how the choice of initialization and optimizer can influence this behavior.

Citations

• No bad local minima: Data independent training error guarantees for\n multilayer neural networks
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data

Related

• Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data
• When does gradient descent with logistic loss find interpolating two-layer networks?
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
• Gradient Descent Quantizes ReLU Network Features
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks