Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

2022/06/02 by Étienne Boursier, Loucas Pillaud‐Vivien, Boursier, Etienne +3 · 7 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2206.00939

openalex publication_date 2022/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, a precise description of the gradient flow dynamics of training one-hidden layer ReLU neural networks for the mean squared error at small initialisation. In this setting, despite non-convexity, we show that the gradient flow converges to zero loss and characterise its implicit bias towards minimum variation norm. Furthermore, some interesting phenomena are highlighted: a quantitative description of the initial alignment phenomenon and a proof that the process follows a specific saddle to saddle dynamics.

Cited by

• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• A mathematical theory for understanding when abstract representations emerge in neural networks
• Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region
• Gradient Flow Convergence Guarantee for General Neural Network Architectures
• Learning In-context \pmbn-grams with Transformers: Sub-\pmbn-grams Are Near-stationary Points
• Understanding Learning Dynamics Through Structured Representations
• Structured Transformations for Stable and Interpretable Neural Computation

Related

• Gradient Dynamics of Shallow Univariate ReLU Networks
• Magnitude and Angle Dynamics in Training Single ReLU Neurons
• Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data