Gradient descent for deep equilibrium single-index models

2025/11/21 by Sanjit Dandapanthula, Aaditya Ramdas, Dandapanthula, Sanjit +1
Computer Science · Physics and Astronomy · #Advanced Graph Neural Networks #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2511.16976

openalex publication_date 2025/11/21 · openalex created_date 2025/11/25 · openalex updated_date 2026/07/30

Abstract

Deep equilibrium models (DEQs) have recently emerged as a powerful paradigm for training infinitely deep weight-tied neural networks that achieve state of the art performance across many modern machine learning tasks. Despite their practical success, theoretically understanding the gradient descent dynamics for training DEQs remains an area of active research. In this work, we rigorously study the gradient descent dynamics for DEQs in the simple setting of linear models and single-index models, filling several gaps in the literature. We prove a conservation law for linear DEQs which implies that the parameters remain trapped on spheres during training and use this property to show that gradient flow remains well-conditioned for all time. We then prove linear convergence of gradient descent to a global minimizer for linear DEQs and deep equilibrium single-index models under appropriate initialization and with a sufficiently small step size. Finally, we validate our theoretical findings through experiments.

Citations

• Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling
• Reversible Deep Equilibrium Models
• Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
• One-Step Diffusion Distillation via Deep Equilibrium Models
• Deep Equilibrium Based Neural Operators for Steady-State PDEs
• On the Neural Tangent Kernel of Equilibrium Models
• Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
• LION: Implicit Vision Prompt Tuning
• Deep Equilibrium Approaches to Diffusion Models
• MGNNI: Multiscale Graph Neural Networks with Implicit Layers
• On the optimization and generalization of overparameterized implicit neural networks
• Deep equilibrium networks are sensitive to initialization statistics
• Global Convergence of Over-parameterized Deep Equilibrium Models
• On Training Implicit Models
• Stabilizing Equilibrium Models by Jacobian Regularization
• Learning a Single Neuron with Bias Using Gradient Descent
• JFB: Jacobian-Free Backpropagation for Implicit Networks
• Deep Equilibrium Architectures for Inverse Problems in Imaging
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• Lipschitz Bounded Equilibrium Networks
• Monotone operator equilibrium networks
• Multiscale Deep Equilibrium Models
• Learning a Single Neuron with Gradient Methods
• Deep Equilibrium Models
• Implicit Deep Learning
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• How to Escape Saddle Points Efficiently
• The Landscape of Empirical Risk for Non-convex Losses
• When Are Nonconvex Problems Not Scary?
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition

Related

• Global Convergence of Over-parameterized Deep Equilibrium Models
• Efficient Training of Deep Equilibrium Models
• DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows
• Reversible Deep Equilibrium Models
• Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models