How Implicit Regularization of ReLU Neural Networks Characterizes the\n Learned Function -- Part I: the 1-D Case of Two Layers with Random First\n Layer

2019/11/07 by Jakob Heiss, Heiss, Jakob, Josef Teichmann +3 · 1 citation
Computer Science · Mathematics · Physics and Astronomy · #41Axx #68Q32 #68T05 #93Exx #FOS: Computer and information sciences #FOS: Mathematics #G.3 #I.2.6 #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Model Reduction and Neural Networks #Numerical Analysis (math.NA) #Numerical methods in inverse problems #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1911.02903

openalex publication_date 2019/11/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we consider one dimensional (shallow) ReLU neural networks in\nwhich weights are chosen randomly and only the terminal layer is trained.\nFirst, we mathematically show that for such networks L2-regularized regression\ncorresponds in function space to regularizing the estimate's second derivative\nfor fairly general loss functionals. For least squares regression, we show that\nthe trained network converges to the smooth spline interpolation of the\ntraining data as the number of hidden nodes tends to infinity. Moreover, we\nderive a novel correspondence between the early stopped gradient descent\n(without any explicit regularization of the weights) and the smoothing spline\nregression.\n

Citations

• Adam: A Method for Stochastic Optimization
• Approximation by superpositions of a sigmoidal function
• Implicit Regularization in Deep Learning
• Gradient Descent Quantizes ReLU Network Features
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Theory IIIb: Generalization in Deep Networks
• How do infinite width bounded norm networks look in function space?
• Implicit Regularization in Over-parameterized Neural Networks
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural\n Networks
• Gradient Dynamics of Shallow Univariate ReLU Networks

Cited by

• Gradient Descent as a Shrinkage Operator for Spectral Bias

Related

• Implicit Regularization in ReLU Networks with the Square Loss
• How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer
• Implicit Regularization in Over-parameterized Neural Networks
• Regularization by Misclassification in ReLU Neural Networks
• Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process