Dropout is a special case of the stochastic delta rule: faster and more\n accurate deep learning

2018/08/10 by Noah Frazier-Logue, Frazier-Logue, Noah, Stephen José Hanson +1 · 2 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1808.03578

openalex publication_date 2018/08/10 · openalex created_date 2022/09/15 · openalex updated_date 2026/07/28

Abstract

Multi-layer neural networks have lead to remarkable performance on many kinds\nof benchmark tasks in text, speech and image processing. Nonlinear parameter\nestimation in hierarchical models is known to be subject to overfitting and\nmisspecification. One approach to these estimation and related problems (local\nminima, colinearity, feature discovery etc.) is called Dropout (Hinton, et al\n2012, Baldi et al 2016). The Dropout algorithm removes hidden units according\nto a Bernoulli random variable with probability p prior to each update,\ncreating random "shocks" to the network that are averaged over updates. In this\npaper we will show that Dropout is a special case of a more general model\npublished originally in 1990 called the Stochastic Delta Rule, or SDR (Hanson,\n1990). SDR redefines each weight in the network as a random variable with mean\n\μ_w_ij and standard deviation \σ_w_ij. Each weight random\nvariable is sampled on each forward activation, consequently creating an\nexponential number of potential networks with shared weights. Both parameters\nare updated according to prediction error, thus resulting in weight noise\ninjections that reflect a local history of prediction error and local model\naveraging. SDR therefore implements a more sensitive local gradient-dependent\nsimulated annealing per weight converging in the limit to a Bayes optimal\nnetwork. Tests on standard benchmarks (CIFAR) using a modified version of\nDenseNet shows the SDR outperforms standard Dropout in test error by approx.\n17 % with DenseNet-BC 250 on CIFAR-100 and approx. 12-14 % in smaller\nnetworks. We also show that SDR reaches the same accuracy that Dropout attains\nin 100 epochs in as few as 35 epochs.\n

Cited by

• Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
• Non-Determinism in Neural Networks for Adversarial Robustness

Related

• Variational Dropout Sparsifies Deep Neural Networks
• Dropout as a Structured Shrinkage Prior
• Understanding Dropout as an Optimization Trick
• Continuous Dropout
• Dropout in Training Neural Networks: Flatness of Solution and Noise Structure