Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing\n its Gradient Estimator Bias

2020/06/06 by Axel Laborieux, Laborieux, Axel, Maxence Ernoult +9 · 16 citations
Computer Science · Engineering · Mathematics · Neuroscience · #Advanced Memory and Neural Computing #Algorithm #Artificial intelligence #Artificial neural network #Backpropagation #Computer science #Deep learning #Domain Adaptation and Few-Shot Learning #Estimator #Locality #MNIST database #Mathematics #Neural dynamics and brain function #Physics #Scalability #Scale (ratio) #Scaling #cs.NE

paper · pdf · doi:10.48550/arxiv.2006.03824

published in arXiv (Cornell University) (Cornell University)

arxiv created 2020/06/06 · openalex publication_date 2020/06/06 · arxiv updated 2020/06/09 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Equilibrium Propagation (EP) is a biologically-inspired algorithm for\nconvergent RNNs with a local learning rule that comes with strong theoretical\nguarantees. The parameter updates of the neural network during the credit\nassignment phase have been shown mathematically to approach the gradients\nprovided by Backpropagation Through Time (BPTT) when the network is\ninfinitesimally nudged toward its target. In practice, however, training a\nnetwork with the gradient estimates provided by EP does not scale to visual\ntasks harder than MNIST. In this work, we show that a bias in the gradient\nestimate of EP, inherent in the use of finite nudging, is responsible for this\nphenomenon and that cancelling it allows training deep ConvNets by EP. We show\nthat this bias can be greatly reduced by using symmetric nudging (a positive\nnudging and a negative one). We also generalize previous EP equations to the\ncase of cross-entropy loss (by opposition to squared error). As a result of\nthese advances, we are able to achieve a test error of 11.7% on CIFAR-10 by EP,\nwhich approaches the one achieved by BPTT and provides a major improvement with\nrespect to the standard EP approach with same-sign nudging that gives 86% test\nerror. We also apply these techniques to train an architecture with asymmetric\nforward and backward connections, yielding a 13.2% test error. These results\nhighlight EP as a compelling biologically-plausible approach to compute error\ngradients in deep neural networks.\n

Cited by

• Learning Dynamics in Memristor-Based Equilibrium Propagation
• Equilibrium Propagation Without Limits
• Learning at the Speed of Physics: Equilibrium Propagation on Oscillator Ising Machines
• Scaling Equilibrium Propagation to Deeper Neural Network Architectures
• Can Local Learning Match Self-Supervised Backpropagation?
• Self-Organising Memristive Networks as Physical Learning Systems
• Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs
• Toward Lifelong Learning in Equilibrium Propagation: Sleep-like and Awake Rehearsal for Enhanced Stability
• Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections
• Towards the Training of Deeper Predictive Coding Neural Networks
• Equilibrium Propagation for Dissipative Dynamics
• Learning long range dependencies through time reversal symmetry breaking
• Equilibrium Propagation for Learning in Lagrangian Dynamical Systems
• How to Train an Oscillator Ising Machine using Equilibrium Propagation
• Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias
• Training and synchronizing oscillator networks with Equilibrium Propagation

Related

• Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias
• Scaling SNNs Trained Using Equilibrium Propagation to Convolutional Architectures
• Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs
• Scaling Equilibrium Propagation to Deeper Neural Network Architectures
• Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input