Deep Learning without Weight Transport

2019/04/10 by Mohamed Akrout, Akrout, Mohamed, Collin Wilson +7 · 16 citations
Computer Science · Engineering · Mathematics · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #CCD and CMOS Imaging Sensors #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1904.05391

Accepted for the Conference on Neural Information Processing Systems (NeurIPS) 2019

openalex publication_date 2019/04/10 · arxiv created 2020/01/09 · arxiv updated 2020/01/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Current algorithms for deep learning probably cannot run in the brain because they rely on weight transport, where forward-path neurons transmit their synaptic weights to a feedback path, in a way that is likely impossible biologically. An algorithm called feedback alignment achieves deep learning without weight transport by using random feedback weights, but it performs poorly on hard visual-recognition tasks. Here we describe two mechanisms - a neural circuit called a weight mirror and a modification of an algorithm proposed by Kolen and Pollack in 1994 - both of which let the feedback path learn appropriate synaptic weights quickly and accurately even in large networks, without weight transport or complex wiring.Tested on the ImageNet visual-recognition task, these mechanisms outperform both feedback alignment and the newer sign-symmetry method, and nearly match backprop, the standard algorithm of deep learning, which uses weight transport.

Citations

• The Self-Tuning Neuron: Synaptic Scaling of Excitatory Synapses
• ImageNet Large Scale Visual Recognition Challenge
• Backpropagation Applied to Handwritten Zip Code Recognition
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Unsupervised Learning by Competing Hidden Units
• Feedback alignment in deep convolutional networks
• TF-Replicator: Distributed Machine Learning for Researchers
• Understanding Reuse, Performance, and Hardware Cost of DNN Dataflows: A Data-Centric Approach Using MAESTRO
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Structural Plasticity as Active Inference: A Biologically-Inspired Architecture for Homeostatic Control
• Feedback Alignment Meets Low-Rank Manifolds: A Structured Recipe for Local Learning
• Reconstructing Backpropagation from Forward Fluctuations in Noise-modulated Neural Networks
• Does Feedback Alignment Work at Biological Timescales?
• Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticity
• Can Local Learning Match Self-Supervised Backpropagation?
• Forward Only Learning for Orthogonal Neural Networks of any Depth
• MAN++: Scaling Momentum Auxiliary Network for Supervised Local Learning in Vision Tasks
• Learning From the Past with Cascading Eligibility Traces
• Loss Landscapes of Regularized Linear Autoencoders
• Learning to solve the credit assignment problem
• One-Time Soft Alignment Enables Resilient Learning without Weight Transport
• Meta-Learning Bidirectional Update Rules
• GAIT-prop: A biologically plausible learning rule derived from backpropagation of error
• Ubiquity of Emergent Hebbian Dynamics in Regularized Learning
• Frozen Backpropagation: Relaxing Weight Symmetry in Deep Spiking Neural Networks
• Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data

Related

• Deep Learning without Weight Symmetry
• Feedback alignment in deep convolutional networks
• Direct Feedback Alignment with Sparse Connections for Local Learning
• Random feedback weights support learning in deep neural networks
• Biologically-plausible learning algorithms can scale to large datasets