A Random Matrix Approach to Neural Networks

2017/02/17 by Louart, Cosme, Liao, Zhenyu, Couillet, Romain · 10 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Probability (math.PR)

paper · doi:10.48550/arxiv.1702.05419

Abstract

This article studies the Gram random matrix model G=\frac1TΣ^{\rm T}Σ, Σ=σ(WX), classically found in the analysis of random feature maps and random neural networks, where X=[x₁,…,x_T]∈\mathbb R^{p× T} is a (data) matrix of bounded norm, W∈\mathbb R^{n× p} is a matrix of independent zero-mean unit variance entries, and σ:\mathbb R→\mathbb R is a Lipschitz continuous (activation) function --- σ(WX) being understood entry-wise. By means of a key concentration of measure lemma arising from non-asymptotic random matrix arguments, we prove that, as n,p,T grow large at the same rate, the resolvent Q=(G+γI_T)^-1, for γ>0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ=\fracTn\mathbb E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.

Cited by

• Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
• From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
• Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networks
• A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyond
• Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix
• Eigenvector overlaps of sample covariance matrices with intersecting time periods
• Incorporating priors in learning: a random matrix study under a teacher-student framework
• Two ways to knowledge?
• Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling
• Spectral distribution of sparse Gaussian Ensembles of Real Asymmetric Matrices

Related

• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Random matrix theory and the loss surfaces of neural networks
• On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case
• Analysis of One-Hidden-Layer Neural Networks via the Resolvent Method
• A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities