Gradient Dynamics of Shallow Univariate ReLU Networks

2019/06/18 by Francis Williams, Williams, Francis, Matthew Trager +9 · 5 citations
Computer Science · Engineering · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Numerical methods in engineering #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1906.07842

openalex publication_date 2019/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a theoretical and empirical study of the gradient dynamics of overparameterized shallow ReLU networks with one-dimensional input, solving least-squares interpolation. We show that the gradient dynamics of such networks are determined by the gradient flow in a non-redundant parameterization of the network function. We examine the principal qualitative features of this gradient flow. In particular, we determine conditions for two learning regimes:kernel and adaptive, which depend both on the relative magnitude of initialization of weights in different layers and the asymptotic behavior of initialization coefficients in the limit of large network widths. We show that learning in the kernel regime yields smooth interpolants, minimizing curvature, and reduces to cubic splines for uniform initializations. Learning in the adaptive regime favors instead linear splines, where knots cluster adaptively at the sample points.

Citations

• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• Gradient Descent Quantizes ReLU Network Features
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• On the Inductive Bias of Neural Tangent Kernels

Cited by

• What Can One Expect When Solving PDEs Using Shallow Neural Networks?
• From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
• When Does Preconditioning Help or Hurt Generalization?
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization

Related

• Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
• Optimization and Generalization of Gradient Descent for Shallow ReLU Networks with Minimal Width
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• Gradient Descent Quantizes ReLU Network Features
• Magnitude and Angle Dynamics in Training Single ReLU Neurons