Mathematical Foundations of Neural Tangents and Infinite-Width Networks

2025/12/09 by Mysore, Rachana, Girish, Preksha, Jayaram, Kavitha +3 · 1 citation
Computer Science · Physics and Astronomy · #60B20 #68T05 #68T07 #FOS: Computer and information sciences #G.3 #Generative Adversarial Networks and Image Synthesis #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques

paper · doi:10.48550/arxiv.2512.08264

openalex publication_date 2025/12/09 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture integrating Fourier feature embeddings, residual connections with layerwise scaling, and stochastic depth to enable rigorous analysis of kernel evolution during training. Our theoretical contributions include deriving bounds on NTK dynamics, characterizing eigenvalue evolution, and linking spectral properties to generalization and optimization stability. Empirical results on synthetic and benchmark datasets validate the predicted kernel behavior and demonstrate improved training stability and generalization. This work provides a comprehensive framework bridging infinite-width theory and practical deep-learning architectures.

Citations

• Eigenvalue distribution of the Neural Tangent Kernel in the quadratic scaling
• Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability
• On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains
• On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model
• On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models
• Feature Learning in Infinite-Width Neural Networks
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• The asymptotic spectrum of the Hessian of DNN throughout training

Cited by

• A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel

Related

• Finite-Width Neural Tangent Kernels from Feynman Diagrams
• On the infinite width limit of neural networks with a standard\n parameterization
• Neural Tangent Kernel: A Survey
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy