Adaptive Normalized Risk-Averting Training For Deep Neural Networks

2015/06/08 by Zhiguang Wang, Tim Oates, Wang, Zhiguang +3 · 1 citation
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1506.02690

openalex publication_date 2015/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper proposes a set of new error criteria and learning approaches, Adaptive Normalized Risk-Averting Training (ANRAT), to attack the non-convex optimization problem in training deep neural networks (DNNs). Theoretically, we demonstrate its effectiveness on global and local convexity lower-bounded by the standard L_p-norm error. By analyzing the gradient on the convexity index λ, we explain the reason why to learn λ adaptively using gradient descent works. In practice, we show how this method improves training of deep neural networks to solve visual recognition tasks on the MNIST and CIFAR-10 datasets. Without using pretraining or other tricks, we obtain results comparable or superior to those reported in recent literature on the same tasks using standard ConvNets + MSE/cross entropy. Performance on deep/shallow multilayer perceptrons and Denoised Auto-encoders is also explored. ANRAT can be combined with other quasi-Newton training methods, innovative network variants, regularization techniques and other specific tricks in DNNs. Other than unsupervised pretraining, it provides a new perspective to address the non-convex optimization problem in DNNs.

Citations

• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• On Tilted Losses in Machine Learning: Theory and Applications

Related

• Adaptively Solving the Local-Minimum Problem for Deep Neural Networks
• Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks
• Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding
• Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods