Regularization by Misclassification in ReLU Neural Networks

2021/11/03 by Elisabetta Cornacchia, Cornacchia, Elisabetta, Jan Hązła +5
Chemistry · Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Algorithm #Artificial intelligence #Artificial neural network #Chemistry #Computer science #Deep neural networks #FOS: Computer and information sciences #Fraction (chemistry) #Machine Learning (cs.LG) #Machine Learning and Data Classification #Mathematics #Neural and Evolutionary Computing (cs.NE) #Noise (video) #Pattern recognition (psychology) #Regularization (linguistics) #Smoothing #Statistics #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2111.02154

arxiv created 2021/11/03 · openalex publication_date 2021/11/03 · arxiv updated 2021/11/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the implicit bias of ReLU neural networks trained by a variant of SGD where at each step, the label is changed with probability p to a random label (label smoothing being a close variant of this procedure). Our experiments demonstrate that label noise propels the network to a sparse solution in the following sense: for a typical input, a small fraction of neurons are active, and the firing pattern of the hidden layers is sparser. In fact, for some instances, an appropriate amount of label noise does not only sparsify the network but further reduces the test error. We then turn to the theoretical analysis of such sparsification mechanisms, focusing on the extremal case of p=1. We show that in this case, the network withers as anticipated from experiments, but surprisingly, in different ways that depend on the learning rate and the presence of bias, with either weights vanishing or neurons ceasing to fire.

Citations

• A Variant of Azuma's Inequality for Martingales with Subgaussian Tails
• Adding noise to the input of a model trained with a regularized objective
• L2 Regularization versus Batch and Weight Normalization
• Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
• Poly-time universality and limitations of deep learning
• What Do Neural Networks Learn When Trained With Random Labels?

Related

• How Implicit Regularization of ReLU Neural Networks Characterizes the\n Learned Function -- Part I: the 1-D Case of Two Layers with Random First\n Layer
• Implicit Regularization in ReLU Networks with the Square Loss
• Training shallow ReLU networks on noisy data using hinge loss: when do we overfit and is it benign?
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• On Sparsity in Overparametrised Shallow ReLU Networks