Meta-Learning Bidirectional Update Rules

2021/04/10 by Mark Sandler, M. Sandler, Max Vladymyrov +14
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2104.04657

ICML 2021, 17 pages

openalex publication_date 2021/04/10 · arxiv created 2021/06/11 · arxiv updated 2021/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule. In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional "genome". We show that such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.

Citations

• Towards Biologically Plausible Deep Learning
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• A Stochastic Approximation Method
• Gradient-based learning applied to document recognition
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Direct Feedback Alignment Provides Learning in Deep Neural Networks
• EMNIST: an extension of MNIST to handwritten letters
• Learned Optimizers that Scale and Generalize
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neuromodulated Learning in Deep Neural Networks
• Deep Learning without Weight Transport
• Deep Residual Learning for Image Recognition
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Reverse engineering learned optimizers reveals known and novel mechanisms
• Continual Learning with Deep Artificial Neurons

Related

• Meta-Learning Update Rules for Unsupervised Representation Learning
• A contrastive rule for meta-learning
• Finding online neural update rules by learning to remember
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Learning to Learn with Feedback and Local Plasticity