Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization

2017/10/14 by Hyeonwoo Noh, Tackgeun You, Noh, Hyeonwoo +5 · 11 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.1710.05179

NIPS 2017 camera ready

openalex publication_date 2017/10/14 · arxiv created 2017/11/09 · arxiv updated 2017/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.

Citations

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Auto-Encoding Variational Bayes
• Improved Dropout for Shallow and Deep Learning
• Wide Residual Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Networks with Stochastic Depth
• VQA: Visual Question Answering
• Learning Deconvolution Network for Semantic Segmentation
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• CoSeNet: A Novel Approach for Optimal Segmentation of Correlation Matrices
• Evaluation of Wafer-Scale SOT-MRAM for Analog Crossbar Array Applications
• Deep Neural Networks Inspired by Differential Equations
• VISP: Volatility Informed Stochastic Projection for Adaptive Regularization
• A Self-Ensemble Inspired Approach for Effective Training of Binary-Weight Spiking Neural Networks
• Representation learning with a transformer by contrastive learning for money laundering detection
• Cross-regularization: Adaptive Model Complexity through Validation Gradients
• Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
• Relevance-driven Input Dropout: an Explanation-guided Regularization Technique
• PANDA: AdaPtive Noisy Data Augmentation for Regularization of Undirected Graphical Models
• Spectral methods: crucial for machine learning, natural for quantum computers?

Related

• Neural Network Regularization via Robust Weight Factorization
• Learning Deep Networks from Noisy Labels with Dropout Regularization
• Bridgeout: stochastic bridge regularization for deep neural networks
• PerNodeDrop: A Method Balancing Specialized Subnets and Regularization in Deep Neural Networks
• Explicit Regularization via Regularizer Mirror Descent