Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weights

2025/07/24 by Takeuchi, Jun'ichi, Takeishi, Yoshinari, Murata, Noboru +3
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2507.18555

Abstract

Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weight are argued. We discuss the relation between both notions as a linear transformation and show that spectral decomposition of NTK with concrete forms of eigenfunctions with major eigenvalues. We also obtain an approximation formula of the functions presented by the 2-layer neural networks.

Citations

• Function Forms of Simple ReLU Networks with Random Hidden Weights
• Risk Bounds on MDL Estimators for Linear Regression Models with Application to Simple ReLU Neural Networks
• Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks
• On the Inductive Bias of Neural Tangent Kernels
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Related

• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks
• Function Forms of Simple ReLU Networks with Random Hidden Weights
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Generalization Ability of Wide Neural Networks on ℝ