Reactivation: Empirical NTK Dynamics Under Task Shifts

2025/07/21 by Yuzhi Liu, Liu, Yuzhi, Zixuan Chen +7
Computer Science · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2507.16039

openalex publication_date 2025/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Neural Tangent Kernel (NTK) offers a powerful tool to study the functional dynamics of neural networks. In the so-called lazy, or kernel regime, the NTK remains static during training and the network function is linear in the static neural tangents feature space. The evolution of the NTK during training is necessary for feature learning, a key driver of deep learning success. The study of the NTK dynamics has led to several critical discoveries in recent years, in generalization and scaling behaviours. However, this body of work has been limited to the single task setting, where the data distribution is assumed constant over time. In this work, we present a comprehensive empirical analysis of NTK dynamics in continual learning, where the data distribution shifts over time. Our findings highlight continual learning as a rich and underutilized testbed for probing the dynamics of neural training. At the same time, they challenge the validity of static-kernel approximations in theoretical treatments of continual learning, even at large scale.

Citations

• On the Cone Effect in the Learning Dynamics
• A Theory of Neural Tangent Kernel Alignment and Its Influence on Training
• Gradient Descent on Neural Networks Typically Occurs at the Edge of\n Stability
• Feature Learning in Infinite-Width Neural Networks
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Similarity of Neural Network Representations Revisited
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Related

• Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective
• The Recurrent Neural Tangent Kernel
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Limitations of the NTK for Understanding Generalization in Deep Learning