2017/06/13 by Brendan Ruff, Ruff, Brendan
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and ELM #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1706.03907
openalex publication_date 2017/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Training a deep convolutional neural net typically starts with a random\ninitialisation of all filters in all layers which severely reduces the forward\nsignal and back-propagated error and leads to slow and sub-optimal training.\nTechniques that counter that focus on either increasing the signal or\nincreasing the gradients adaptively but the model behaves very differently at\nthe beginning of training compared to later when stable pathways through the\nnet have been established. To compound this problem the effective minibatch\nsize varies greatly between layers at different depths and between individual\nfilters as activation sparsity typically increases with depth leading to a\nreduction in effective learning rate since gradients may superpose rather than\nadd and this further compounds the covariate shift problem as deeper neurons\nare less able to adapt to upstream shift.\n Proposed here is a method of automatic gain control of the signal built into\neach convolutional neuron that achieves equivalent or superior performance than\nbatch normalisation and is compatible with single sample or minibatch gradient\ndescent. The same model is used both for training and inference.\n The technique comprises a scaled per sample map mean subtraction from the raw\nconvolutional filter output followed by scaling of the difference.\n