2021/10/27 by Paul Haider, Haider, Paul, Benjamin Ellenberger +9 · 1 voice · 5 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · Neuroscience · #Advanced Memory and Neural Computing #Artificial Intelligence (cs.AI) #B.8.1 #F.1.1 #FOS: Biological sciences #FOS: Computer and information sciences #FOS: Electrical engineering #I.2.6 #I.5.1 #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Neural dynamics and brain function #Neurons and Cognition (q-bio.NC) #Signal Processing (eess.SP) #cs.AI #cs.LG #cs.NE #eess.SP #electronic engineering #information engineering #q-bio.NC
paper · pdf · doi:10.48550/arxiv.2110.14549
Accepted for publication in Advances in Neural Information Processing Systems 34 (NeurIPS 2021); 13 pages, 4 figures; 10 pages of supplementary material, 1 supplementary figure
arxiv created 2021/10/27 · openalex publication_date 2021/10/27 · arxiv published 2021/10/27 · arxiv updated 2021/10/28 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
The response time of physical computational elements is finite, and neurons are no exception. In hierarchical models of cortical networks each layer thus introduces a response lag. This inherent property of physical dynamical systems results in delayed processing of stimuli and causes a timing mismatch between network output and instructive signals, thus afflicting not only inference, but also learning. We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential. This principle enables quasi-instantaneous inference independent of network depth and avoids the need for phased plasticity or computationally expensive network relaxation phases. We jointly derive disentangled neuron and synapse dynamics from a prospective energy function that depends on a network's generalized position and momentum. The resulting model can be interpreted as a biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time, leaky neuronal dynamics and continuously active, local plasticity. We demonstrate successful learning of standard benchmark datasets, achieving competitive performance using both fully-connected and convolutional architectures, and show how our principle can be applied to detailed models of cortical microcircuitry. Furthermore, we study the robustness of our model to spatio-temporal substrate imperfections to demonstrate its feasibility for physical realization, be it in vivo or in silico.