The effect of Leaky ReLUs on the training and generalization of overparameterized networks

2024/02/19 by Yinglong Guo, Guo, Yinglong, Shaohan Li +3 · 1 citation
Computer Science · #FOS: Computer and information sciences #Face and Expression Recognition #Fuzzy Logic and Control Systems #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2402.11942

openalex publication_date 2024/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, α. We show that α=-1, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.

Cited by

• Stochastic activations

Related

• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Architecture independent generalization bounds for overparametrized deep ReLU networks
• Benign overfitting in leaky ReLU networks with moderate input dimension
• Learning ReLU networks to high uniform accuracy is intractable
• From Tempered to Benign Overfitting in ReLU Neural Networks