Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs

2023/06/10 by Dmitry Chistikov, Chistikov, Dmitry, Matthias Englert +3 · 1 citation
Computer Science · Physics and Astronomy · #68Q32 #68T07 #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Machine Learning and ELM #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2306.06479

openalex publication_date 2023/06/10 · openalex created_date 2023/06/14 · openalex updated_date 2026/07/28

Abstract

We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.

Cited by

• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data

Related

• Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks
• Training invariances and the low-rank phenomenon: beyond linear networks
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
• Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs