On the Neural Tangent Kernel of Equilibrium Models

2023/10/21 by Feng, Zhili, Kolter, J. Zico · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2310.14062

Abstract

This work studies the neural tangent kernel (NTK) of the deep equilibrium (DEQ) model, a practical ``infinite-depth'' architecture which directly computes the infinite-depth limit of a weight-tied network via root-finding. Even though the NTK of a fully-connected neural network can be stochastic if its width and depth both tend to infinity simultaneously, we show that contrarily a DEQ model still enjoys a deterministic NTK despite its width and depth going to infinity at the same time under mild conditions. Moreover, this deterministic NTK can be found efficiently via root-finding.

Cited by

• Gradient descent for deep equilibrium single-index models

Related

• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Mathematical Foundations of Neural Tangents and Infinite-Width Networks
• Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
• Revisiting the Neural Tangent Kernel: the role of large width and depth
• Neural Tangent Kernel: A Survey