Tensor Programs II: Neural Tangent Kernel for Any Architecture

2020/06/25 by Greg Yang, Yang, Greg · 27 citations
Computer Science · Mathematics · Physics and Astronomy · #Advanced Neural Network Applications #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #Stochastic Gradient Optimization Techniques #Tensor decomposition and applications #cond-mat.dis-nn #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.14548

11 pages of main text. 60 pages total. August 2020: Fixed "BP-like" definition

openalex publication_date 2020/06/25 · arxiv created 2020/11/30 · arxiv updated 2020/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. We demonstrate how to calculate this limit. In prior literature, the heuristic study of neural network gradients often assumes every weight matrix used in forward propagation is independent from its transpose used in backpropagation (Schoenholz et al. 2017). This is known as the *gradient independence assumption (GIA)*. We identify a commonly satisfied condition, which we call *Simple GIA Check*, such that the NTK limit calculation based on GIA is correct. Conversely, when Simple GIA Check fails, we show GIA can result in wrong answers. Our material here presents the NTK results of Yang (2019a) in a friendly manner and showcases the *tensor programs* technique for understanding wide neural networks. We provide reference implementations of infinite-width NTKs of recurrent neural network, transformer, and batch normalization at https://github.com/thegregyang/NTK4A.

Citations

• Deep Convolutional Networks on Graph-Structured Data
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Cognitron: A self-organizing multilayered neural network
• Long Short-Term Memory
• Gradient-based learning applied to document recognition
• Semi-Supervised Classification with Graph Convolutional Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Towards Characterizing Divergence in Deep Q-Learning
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Asymptotics of Wide Networks from Feynman Diagrams
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Infinite attention: NNGP and NTK for deep attention networks

Cited by

• Feature Learning Dynamics in Infinite-Depth Neural Networks
• Mathematical Foundations of Neural Tangents and Infinite-Width Networks
• Dynamical Implicit Neural Representations
• Fermions and Supersymmetry in Neural Network Field Theories
• Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains
• When Bias Pretends to Be Truth: How Spurious Correlations Undermine Hallucination Detection in LLMs
• Scaling Laws and In-Context Learning: A Unified Theoretical Framework
• A mathematical theory for understanding when abstract representations emerge in neural networks
• Viability of perturbative expansion for quantum field theories on neurons
• Reactivation: Empirical NTK Dynamics Under Task Shifts
• Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens
• Decoupled Relative Learning Rate Schedules
• Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks
• The Importance of Being Lazy: Scaling Limits of Continual Learning
• Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Variational Deep Learning via Implicit Regularization
• LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs
• ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
• Scaling Diffusion Transformers Efficiently via μP
• Just One Layer Norm Guarantees Stable Extrapolation
• Feature Learning in Infinite-Width Neural Networks
• Tensor Programs III: Neural Matrix Laws
• Implicit bias of deep linear networks in the large learning rate phase
• Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
• Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
• Nonperturbative renormalization for the neural network-QFT correspondence

Related

• Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Tensor Programs III: Neural Matrix Laws
• Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit