A Gradient Estimator for Time-Varying Electrical Networks with\n Non-Linear Dissipation

2021/03/08 by Jack D. Kendall, Jack Kendall, Kendall, Jack · 1 citation
Computer Science · Engineering · Mathematics · Physics and Astronomy · #Advanced Memory and Neural Computing #Applied mathematics #Artificial Intelligence (cs.AI) #Artificial intelligence #Artificial neural network #Biological neural network #Capacitor #Computer science #Control theory (sociology) #Dissipation #Dissipative system #Electrical network #Electronic engineering #Engineering #Estimator #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematics #Memristor #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Nonlinear system #Physics #Resistor #Topology (electrical circuits) #cs.AI #cs.LG #cs.NE #stochastic dynamics and bifurcation

paper · pdf · doi:10.48550/arxiv.2103.05636

12 pages, 0 figures

openalex publication_date 2021/03/08 · arxiv created 2021/03/09 · arxiv updated 2021/03/11 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

We propose a method for extending the technique of equilibrium propagation\nfor estimating gradients in fixed-point neural networks to the more general\nsetting of directed, time-varying neural networks by modeling them as\nelectrical circuits. We use electrical circuit theory to construct a Lagrangian\ncapable of describing deep, directed neural networks modeled using nonlinear\ncapacitors and inductors, linear resistors and sources, and a special class of\nnonlinear dissipative elements called fractional memristors. We then derive an\nestimator for the gradient of the physical parameters of the network, such as\nsynapse conductances, with respect to an arbitrary loss function. This\nestimator is entirely local, in that it only depends on information locally\navailable to each synapse. We conclude by suggesting methods for extending\nthese results to networks of biologically plausible neurons, e.g.\nHodgkin-Huxley neurons.\n

Citations

• Generative Adversarial Networks
• Playing Atari with Deep Reinforcement Learning
• The classical mechanics of non-conservative systems
• A quantitative description of membrane current and its application to conduction and excitation in nerve
• New types of deep neural network learning for speech recognition and related applications: an overview
• On the loss landscape of a class of deep neural networks with no bad local valleys
• A deep learning framework for neuroscience
• Training End-to-End Analog Neural Networks with Equilibrium Propagation

Cited by

• Equilibrium Propagation for Dissipative Dynamics

Related

• Training End-to-End Analog Neural Networks with Equilibrium Propagation
• Gradient modelling of memristive systems
• A deep learning theory for neural networks grounded in physics
• A Computational Approach for the inverse problem of neuronal conductances determination
• An Operator-Theoretic Framework to Simulate Neuromorphic Circuits