Gradient Normalization & Depth Based Decay For Deep Learning

2017/12/10 by Robert Kwiatkowski, Oscar Chang, Kwiatkowski, Robert +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1712.03607

Results seemed more promising at the time

arxiv created 2018/02/28 · arxiv updated 2018/03/01

Abstract

In this paper we introduce a novel method of gradient normalization and decay with respect to depth. Our method leverages the simple concept of normalizing all gradients in a deep neural network, and then decaying said gradients with respect to their depth in the network. Our proposed normalization and decay techniques can be used in conjunction with most current state of the art optimizers and are a very simple addition to any network. This method, although simple, showed improvements in convergence time on state of the art networks such as DenseNet and ResNet on image classification tasks, as well as on an LSTM for natural language processing tasks.

Related

• Backward Gradient Normalization in Deep Neural Networks
• Deep Gradient Boosting -- Layer-wise Input Normalization of Neural Networks
• Weight and Gradient Centralization in Deep Neural Networks
• Insights from Gradient Dynamics: Gradient Autoscaled Normalization
• Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks