Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks

2021/09/20 by Zhichao Wang, Yizhe Zhu, Wang, Zhichao +1 · 5 citations
Computer Science · #60B20 (Primary) 68T07 #62J07 (Secondary) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Matrix Theory and Algorithms #Neural Networks and Applications #Probability (math.PR) #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2109.09304

openalex publication_date 2021/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

In this paper, we investigate a two-layer fully connected neural network of the form f(X)=(1)/(√(d₁))\boldsymbola^\top σ(WX), where X∈ℝ^{d₀× n} is a deterministic data matrix, W∈ℝ^{d₁× d₀} and \boldsymbola∈ℝ^d₁ are random Gaussian weights, and σ is a nonlinear activation function. We study the limiting spectral distributions of two empirical kernel matrices associated with f(X): the empirical conjugate kernel (CK) and neural tangent kernel (NTK), beyond the linear-width regime (d₁\asymp n). We focus on the ultra-wide regime, where the width d₁ of the first layer is much larger than the sample size n. Under appropriate assumptions on X and σ, a deformed semicircle law emerges as d₁/n→∞ and n→∞. We first prove this limiting law for generalized sample covariance matrices with some dependency. To specify it for our neural network model, we provide a nonlinear Hanson-Wright inequality that is suitable for neural networks with random weights and Lipschitz activation functions. We also demonstrate non-asymptotic concentrations of the empirical CK and NTK around their limiting kernels in the spectral norm, along with lower bounds on their smallest eigenvalues. As an application, we show that random feature regression induced by the empirical kernel achieves the same asymptotic performance as its limiting kernel regression under the ultra-wide regime. This allows us to calculate the asymptotic training and test errors for random feature regression using the corresponding kernel regression.

Citations

• Limiting spectral distribution of renormalized separable sample covariance matrices whenp/n→0
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• Partial transposition of random states and non-centered semicircular\n distributions
• On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions
• A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
• Stationary Points of Shallow Neural Networks with Quadratic Activation\n Function
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a\n Multi-Scale Theory of Generalization
• Universality Laws for High-Dimensional Learning with Random Features
• Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Asymptotic normality for eigenvalue statistics of a general sample covariance matrix when p/n → ∞ and applications

Cited by

• Low Rank Gradients and Where to Find Them
• A Law of Data Reconstruction for Random Features (and Beyond)
• Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling
• Spectra of high-dimensional sparse random geometric graphs
• Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel

Related

• A Random Matrix Approach to Neural Networks
• Generalization Ability of Wide Neural Networks on ℝ
• Analysis of One-Hidden-Layer Neural Networks via the Resolvent Method
• Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling
• Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks