Why Smooth Stability Assumptions Fail for ReLU Learning

2025/12/26 by Ronald Katende, Katende, Ronald
Computer Science · #Advanced Graph Neural Networks #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · doi:10.48550/arxiv.2512.22055

openalex publication_date 2025/12/26 · openalex created_date 2025/12/30 · openalex updated_date 2026/07/28

Abstract

Stability analyses of modern learning systems are frequently derived under smoothness assumptions that are violated by ReLU-type nonlinearities. In this note, we isolate a minimal obstruction by showing that no uniform smoothness-based stability proxy such as gradient Lipschitzness or Hessian control can hold globally for ReLU networks, even in simple settings where training trajectories appear empirically stable. We give a concrete counterexample demonstrating the failure of classical stability bounds and identify a minimal generalized derivative condition under which stability statements can be meaningfully restored. The result clarifies why smooth approximations of ReLU can be misleading and motivates nonsmooth-aware stability frameworks.

Citations

• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Train faster, generalize better: Stability of stochastic gradient descent

Related

• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
• The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon
• A Learning Stability Profile for Finite-Dimensional Learning Dynamics
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability