The Optimal Condition Number for ReLU Function

2025/04/16 by Yu Xia, Haoyu Zhou, Xia, Yu +1
Computer Science · #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning and ELM #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2504.12194

openalex publication_date 2025/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

ReLU is a widely used activation function in deep neural networks. This paper explores the stability properties of the ReLU map. For any weight matrix \boldsymbolA ∈ ℝ^{m × n} and bias vector \boldsymbolb ∈ ℝ^m at a given layer, we define the condition number β_\boldsymbolA,\boldsymbolb as β_\boldsymbolA,\boldsymbolb = \fracU_\boldsymbolA,\boldsymbolbL_\boldsymbolA,\boldsymbolb, where U_\boldsymbolA,\boldsymbolb and L_\boldsymbolA,\boldsymbolb are the upper and lower Lipschitz constants, respectively. We first demonstrate that for any given \boldsymbolA and \boldsymbolb, the condition number satisfies β_\boldsymbolA,\boldsymbolb ≥ √(2). Moreover, when the weights of the network at a given layer are initialized as random i.i.d. Gaussian variables and the bias term is set to zero, the condition number asymptotically approaches this lower bound. This theoretical finding suggests that Gaussian weight initialization is optimal for preserving distances in the context of random deep neural network weights.

Related

• Near-optimal estimates for the ℓ^p-Lipschitz constants of deep random ReLU neural networks
• Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality
• Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
• Upper and lower bounds for the Lipschitz constant of random neural networks
• Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time