Training DNNs in O(1) memory with MEM-DFA using Random Matrices

2020/12/21 by Tien Dung Chu, Tien Chu, Kamil Mykitiuk +8 · 1 voice · 1 citation
Computer Science · Engineering · Mathematics · #Advanced Memory and Neural Computing #Algorithm #Artificial intelligence #Artificial neural network #Backpropagation #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Constant (computer programming) #FOS: Computer and information sciences #Independence (probability theory) #MNIST database #Machine Learning and ELM #Mathematics #Neural Networks and Applications #Statistics #cs.CV

paper · pdf · doi:10.48550/arxiv.2012.11745

arxiv created 2020/12/21 · openalex publication_date 2020/12/21 · arxiv published 2020/12/21 · arxiv updated 2020/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work presents a method for reducing memory consumption to a constant complexity when training deep neural networks. The algorithm is based on the more biologically plausible alternatives of the backpropagation (BP): direct feedback alignment (DFA) and feedback alignment (FA), which use random matrices to propagate error. The proposed method, memory-efficient direct feedback alignment (MEM-DFA), uses higher independence of layers in DFA and allows avoiding storing at once all activation vectors, unlike standard BP, FA, and DFA. Thus, our algorithm's memory usage is constant regardless of the number of layers in a neural network. The method increases the computational cost only by a constant factor of one extra forward pass. The MEM-DFA, BP, FA, and DFA were evaluated along with their memory profiles on MNIST and CIFAR-10 datasets on various neural network models. Our experiments agree with our theoretical results and show a significant decrease in the memory cost of MEM-DFA compared to the other algorithms.

Citations

• Learning representations by back-propagating errors
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How Important is Weight Symmetry in Backpropagation?
• Random feedback weights support learning in deep neural networks
• Training Deep Nets with Sublinear Memory Cost
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Feedback alignment in deep convolutional networks
• Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs

Cited by

• Convergence Analysis and Implicit Regularization of Feedback Alignment\n for Deep Linear Networks

Discussions

• Training DNN in O(1) Memory Using Random Matrices [hn, 2 points, 0 comments]

Related

• Extension of Direct Feedback Alignment to Convolutional and Recurrent Neural Network for Bio-plausible Deep Learning
• Direct Feedback Alignment with Sparse Connections for Local Learning
• Efficient Convolutional Neural Network Training with Direct Feedback Alignment
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• Principled Training of Neural Networks with Direct Feedback Alignment