2020/06/06 by Axel Laborieux, Laborieux, Axel, Maxence Ernoult +9 · 9 citations
Neuroscience · Engineering · Computer Science · #Neural dynamics and brain function #Advanced Memory and Neural Computing #Domain Adaptation and Few-Shot Learning
paper · pdf · doi:10.48550/arxiv.2006.03824
Equilibrium Propagation (EP) is a biologically-inspired algorithm for\nconvergent RNNs with a local learning rule that comes with strong theoretical\nguarantees. The parameter updates of the neural network during the credit\nassignment phase have been shown mathematically to approach the gradients\nprovided by Backpropagation Through Time (BPTT) when the network is\ninfinitesimally nudged toward its target. In practice, however, training a\nnetwork with the gradient estimates provided by EP does not scale to visual\ntasks harder than MNIST. In this work, we show that a bias in the gradient\nestimate of EP, inherent in the use of finite nudging, is responsible for this\nphenomenon and that cancelling it allows training deep ConvNets by EP. We show\nthat this bias can be greatly reduced by using symmetric nudging (a positive\nnudging and a negative one). We also generalize previous EP equations to the\ncase of cross-entropy loss (by opposition to squared error). As a result of\nthese advances, we are able to achieve a test error of 11.7% on CIFAR-10 by EP,\nwhich approaches the one achieved by BPTT and provides a major improvement with\nrespect to the standard EP approach with same-sign nudging that gives 86% test\nerror. We also apply these techniques to train an architecture with asymmetric\nforward and backward connections, yielding a 13.2% test error. These results\nhighlight EP as a compelling biologically-plausible approach to compute error\ngradients in deep neural networks.\n