Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks

2022/02/11 by Sangmin Lee, Lee, Sangmin, Byeongsu Sim +3
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2202.05510

openalex publication_date 2022/02/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Understanding implicit bias of gradient descent for generalization capability of ReLU networks has been an important research topic in machine learning research. Unfortunately, even for a single ReLU neuron trained with the square loss, it was recently shown impossible to characterize the implicit regularization in terms of a norm of model parameters (Vardi & Shamir, 2021). In order to close the gap toward understanding intriguing generalization behavior of ReLU networks, here we examine the gradient flow dynamics in the parameter space when training single-neuron ReLU networks. Specifically, we discover an implicit bias in terms of support vectors, which plays a key role in why and how ReLU networks generalize well. Moreover, we analyze gradient flows with respect to the magnitude of the norm of initialization, and show that the norm of the learned weight strictly increases through the gradient flow. Lastly, we prove the global convergence of single ReLU neuron for d = 2 case.

Related

• Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias
• Magnitude and Angle Dynamics in Training Single ReLU Neurons
• The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks
• Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data