Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime

2023/10/27 by Sofiia Dubova, Dubova, Sofiia, Yue M. Lu +5 · 4 citations
Mathematics · #15B52 #60B20 #Advanced Algebra and Geometry #Advanced Combinatorial Mathematics #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (stat.ML) #Probability (math.PR) #Random Matrices and Applications

paper · pdf · doi:10.48550/arxiv.2310.18280

openalex publication_date 2023/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We consider certain large random matrices, called random inner-product kernel matrices, which are essentially given by a nonlinear function f applied entrywise to a sample-covariance matrix, f(X^TX), where X ∈ ℝ^{d × N} is random and normalized in such a way that f typically has order-one arguments. We work in the polynomial regime, where N \asymp d^ℓ for some ℓ > 0, not just the linear regime where ℓ = 1. Earlier work by various authors showed that, when the columns of X are either uniform on the sphere or standard Gaussian vectors, and when ℓ is an integer (the linear regime ℓ = 1 is particularly well-studied), the bulk eigenvalues of such matrices behave in a simple way: They are asymptotically given by the free convolution of the semicircular and Marčenko-Pastur distributions, with relative weights given by expanding f in the Hermite basis. In this paper, we show that this phenomenon is universal, holding as soon as X has i.i.d. entries with all finite moments. In the case of non-integer ℓ, the Marčenko-Pastur term disappears (its weight in the free convolution vanishes), and the spectrum is just semicircular.

Cited by

• A general technique for approximating high-dimensional empirical kernel matrices
• On the edge eigenvalues of sparse random geometric graphs
• Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling
• Asymptotic theory of in-context learning by linear attention

Related

• An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalings
• The spectrum of random kernel matrices: universality results for rough and varying kernels
• Extremal Eigenvalues of Random Kernel Matrices with Polynomial Scaling
• The Spectrum of Random Inner-product Kernel Matrices
• The Spectral Norm of Random Inner-Product Kernel Matrices