Decoupled Greedy Learning of CNNs

2019/01/23 by Eugene Belilovsky, Michael Eickenberg, Belilovsky, Eugene +3 · 1 citation
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face and Expression Recognition #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1901.08164

openalex publication_date 2019/01/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A commonly cited inefficiency of neural network training by back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simpler, but more effective, substitute that uses minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization in layers. With the use of a replay buffer we show this approach can be extended to asynchronous settings, where modules can operate with possibly large communication delays. We show theoretically and empirically that this approach converges. Then, we empirically find that it can lead to better generalization than sequential greedy optimization. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Random feedback weights support learning in deep neural networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Wide Residual Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp\n Minima
• Understanding Synthetic Gradients and Decoupled Neural Interfaces
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Asynchronous Decentralized Parallel Stochastic Gradient Descent
• Improved asynchronous parallel optimization analysis for stochastic incremental methods
• Assessing the Scalability of Biologically-Motivated Deep Learning\n Algorithms and Architectures
• Training Neural Networks with Local Error Signals
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis

Related

• Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Training CNNs faster with Dynamic Input and Kernel Downsampling
• Reversible designs for extreme memory cost reduction of CNN training