Are skip connections necessary for biologically plausible learning rules?

2019/12/04 by Daniel Jiwoong Im, Im, Daniel Jiwoong, Rutuja Patil +3
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.2001.01647

openalex publication_date 2019/12/04 · openalex created_date 2020/01/10 · openalex updated_date 2026/07/28

Abstract

Backpropagation is the workhorse of deep learning, however, several other biologically-motivated learning rules have been introduced, such as random feedback alignment and difference target propagation. None of these methods have produced a competitive performance against backpropagation. In this paper, we show that biologically-motivated learning rules with skip connections between intermediate layers can perform as well as backpropagation on the MNIST dataset and are robust to various sets of hyper-parameters.

Citations

• Assessing the Scalability of Biologically-Motivated Deep Learning\n Algorithms and Architectures

Related

• Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathways
• Towards Biologically Plausible Computing: A Comprehensive Comparison
• How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective
• A More Biologically Plausible Local Learning Rule for ANNs
• Biologically-plausible learning algorithms can scale to large datasets