Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment

2020/02/10 by Alexander G. Ororbia, Ankur Mali, Ororbia, Alexander +5
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.2002.03911

openalex publication_date 2020/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Training deep neural networks on large-scale datasets requires significant hardware resources whose costs (even on cloud platforms) put them out of reach of smaller organizations, groups, and individuals. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. Furthermore, it requires researchers to continually develop various tricks, such as specialized weight initializations and activation functions, in order to ensure a stable parameter optimization. Our goal is to seek an effective, neuro-biologically-plausible alternative to backprop that can be used to train deep networks. In this paper, we propose a gradient-free learning procedure, recursive local representation alignment, for training large-scale neural architectures. Experiments with residual networks on CIFAR-10 and the large benchmark, ImageNet, show that our algorithm generalizes as well as backprop while converging sooner due to weight updates that are parallelizable and computationally less demanding. This is empirical evidence that a backprop-free algorithm can scale up to larger datasets.

Citations

• Adam: A Method for Stochastic Optimization
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• How Important is Weight Symmetry in Backpropagation?
• On the difficulty of training Recurrent Neural Networks
• Training Products of Experts by Minimizing Contrastive Divergence
• ImageNet Large Scale Visual Recognition Challenge
• Deeply-Supervised Nets
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Decoupled Neural Interfaces using Synthetic Gradients
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Unsupervised learning by competing hidden units
• Greedy Layerwise Learning Can Scale to ImageNet
• Training Neural Networks with Local Error Signals
• Decoupled Greedy Learning of CNNs
• Self-training with Noisy Student improves ImageNet classification

Related

• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks
• How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective
• Pipelined Backpropagation at Scale: Training Large Models without\n Batches
• Deep supervised learning using local errors