Stochastic Function Norm Regularization of Deep Networks

2016/05/30 by Amal Rannen Triki, Matthew B. Blaschko, Triki, Amal Rannen +1
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1605.09085

openalex publication_date 2016/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep neural networks have had an enormous impact on image analysis. State-of-the-art training methods, based on weight decay and DropOut, result in impressive performance when a very large training set is available. However, they tend to have large problems overfitting to small data sets. Indeed, the available regularization methods deal with the complexity of the network function only indirectly. In this paper, we study the feasibility of directly using the L₂ function norm for regularization. Two methods to integrate this new regularization in the stochastic backpropagation are proposed. Moreover, the convergence of these new algorithms is studied. We finally show that they outperform the state-of-the-art methods in the low sample regime on benchmark datasets (MNIST and CIFAR10). The obtained results demonstrate very clear improvement, especially in the context of small sample regimes with data laying in a low dimensional manifold. Source code of the method can be found at \urlhttps://github.com/AmalRT/DNN_Reg.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Auto-Encoding Variational Bayes
• Monte Carlo sampling methods using Markov chains and their applications
• Sharp Minima Can Generalize For Deep Nets
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Related

• Function Norms and Regularization in Deep Networks
• Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach
• Bridgeout: stochastic bridge regularization for deep neural networks
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• DL-Reg: A Deep Learning Regularization Technique using Linear Regression