Radius-margin bounds for deep neural networks

2018/11/03 by Mayank Sharma, Sharma, Mayank, Jayadeva +2
Computer Science · Engineering · #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.1811.01171

openalex publication_date 2018/11/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Explaining the unreasonable effectiveness of deep learning has eluded researchers around the globe. Various authors have described multiple metrics to evaluate the capacity of deep architectures. In this paper, we allude to the radius margin bounds described for a support vector machine (SVM) with hinge loss, apply the same to the deep feed-forward architectures and derive the Vapnik-Chervonenkis (VC) bounds which are different from the earlier bounds proposed in terms of number of weights of the network. In doing so, we also relate the effectiveness of techniques like Dropout and Dropconnect in bringing down the capacity of the network. Finally, we describe the effect of maximizing the input as well as the output margin to achieve an input noise-robust deep architecture.

Citations

• On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularization
• Wide Residual Networks

Related

• On the Depth of Deep Neural Networks: A Theoretical View
• On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond
• Large Margin Deep Networks for Classification
• Effects of the optimisation of the margin distribution on generalisation in deep architectures
• Pairwise Margin Maximization for Deep Neural Networks