Provable Generalization Bounds for Deep Neural Networks with Momentum-Adaptive Gradient Dropout

2025/10/21 by Adeel Safder, Safder, Adeel
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Statistics Theory (math.ST) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2510.18410

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Deep neural networks (DNNs) achieve remarkable performance but often suffer from overfitting due to their high capacity. We introduce Momentum-Adaptive Gradient Dropout (MAGDrop), a novel regularization method that dynamically adjusts dropout rates on activations based on current gradients and accumulated momentum, enhancing stability in non-convex optimization landscapes. To theoretically justify MAGDrop's effectiveness, we derive a non-asymptotic, computable PAC-Bayes generalization bound that accounts for its adaptive nature, achieving up to 29.2% tighter bounds compared to standard approaches by leveraging momentum-driven perturbation control. Empirically, the activation-based MAGDrop achieves competitive performance on MNIST (99.52%) and CIFAR-10 (92.03%), with generalization gaps of 0.48% and 6.52%, respectively. We provide fully reproducible code and numerical computation of our bounds to validate our theoretical claims. Our work bridges theoretical insights and practical advancements, offering a robust framework for enhancing DNN generalization, making it suitable for high-stakes applications.

Citations

• Sharpness-Aware Minimization for Efficiently Improving Generalization
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks
• Fantastic Generalization Measures and Where to Find Them
• Implicit Regularization in Deep Matrix Factorization
• Stronger generalization bounds for deep nets via a compression approach
• Exploring Generalization in Deep Learning
• Spectrally-normalized margin bounds for neural networks
• Attention Is All You Need
• Understanding deep learning requires rethinking generalization
• ImageNet classification with deep convolutional neural networks
• Gradient-based learning applied to document recognition

Related

• Understanding Dropout as an Optimization Trick
• Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks)