Architecture independent generalization bounds for overparametrized deep ReLU networks

2025/04/08 by Bapu, Anandatheertha, Chen, Thomas, Thomas Chen +5
Computer Science · #57R70 #62M45 #Advanced Graph Neural Networks #Analysis of PDEs (math.AP) #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Optimization and Control (math.OC) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2504.05695

openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

We prove that overparametrized neural networks are able to generalize with a test error that is independent of the level of overparametrization, and independent of the Vapnik-Chervonenkis (VC) dimension. We prove explicit bounds that only depend on the metric geometry of the test and training sets, on the regularity properties of the activation function, and on the operator norms of the weights and norms of biases. For overparametrized deep ReLU networks with a training sample size bounded by the input space dimension, we explicitly construct zero loss minimizers without use of gradient descent, and prove a uniform generalization bound that is independent of the network architecture. We perform computational experiments of our theoretical results with MNIST, and obtain agreement with the true test error within a 22 % margin on average.

Related

• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Learning ReLU networks to high uniform accuracy is intractable
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU\n Networks?