SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

2025/05/14 by Julian Kranz, Davide Gallon, Kranz, Julian +5 · 4 citations
Computer Science · Physics and Astronomy · #03C64 #03C98 #26B40 #FOS: Computer and information sciences #FOS: Mathematics #Logic (math.LO) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Neural Networks and Applications #Optimization and Control (math.OC) #Primary 68T05 #Secondary 68T07

paper · pdf · doi:10.48550/arxiv.2505.09572

openalex publication_date 2025/05/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold ε>0 such that the loss value of any gradient flow initialized at most ε above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.

Citations

• Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks
• Stability of first-order methods in tame optimization
• Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation
• Convergence rates for the Adam optimizer
• Learning-rate-free Momentum SGD with Reshuffling Converges in Nonsmooth Nonconvex Optimization
• Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks
• Global stability of first-order methods for coercive tame functions
• Convex and Non-convex Optimization Under Generalized Smoothness
• On the existence of minimizers in shallow residual ReLU neural network optimization landscapes
• Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability
• Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks
• Approximation results for Gradient Descent trained Shallow Neural Networks in 1d
• On Margin Maximization in Linear and ReLU Networks
• A convex function satisfying the Lojasiewicz inequality but failing the gradient conjecture both at zero and infinity
• Neural Network Approximation
• Directional convergence and alignment in deep learning
• Language Models are Few-Shot Learners
• Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation rates
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Gradient Descent Finds Global Minima of Deep Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Topological properties of the set of functions generated by neural\n networks of fixed size
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Stochastic subgradient method converges on tame functions
• Optimal approximation of piecewise smooth functions using deep ReLU\n neural networks
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Error bounds for approximations with deep ReLU networks
• Gaussian Error Linear Units (GELUs)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods
• Proximal alternating minimization and projection methods for nonconvex\n problems. An approach based on the Kurdyka-Lojasiewicz inequality
• Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Łojasiewicz Inequality
• A generalized Vandermonde determinant
• Multilayer feedforward networks with a nonpolynomial activation function can approximate any function
• Multilayer feedforward networks are universal approximators
• A Stochastic Approximation Method
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Gradient-based learning applied to document recognition

Cited by

• Nonconvex Penalized LAD Estimation in Partial Linear Models with DNNs: Asymptotic Analysis and Proximal Algorithms
• Exact Subgraph Isomorphism Network with Mixed L_0,2 Norm Constraint for Predictive Graph Mining
• Deep Learning as the Disciplined Construction of Tame Objects
• PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning

Related

• Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective
• Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks
• Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
• Flat Channels to Infinity in Neural Loss Landscapes
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties