Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

2019/03/31 by Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda +2 · 432 citations
Computer Science · Engineering · Neuroscience · #Advanced Memory and Neural Computing #Artificial intelligence #Artificial neural network #Backpropagation #Computer science #Convolutional neural network #Deep learning #MNIST database #Machine learning #Neural Networks and Reservoir Computing #Neural dynamics and brain function #Pattern recognition (psychology) #Spike (software development) #Spiking neural network #cs.NE

paper · pdf · doi:10.3389/fnins.2020.00119

published in Frontiers in Neuroscience 14, 119 (Frontiers Media) · Chankyu Lee and Syed Shakib Sarwar contributed equally to the work

openalex publication_date 2020/02/28 · arxiv created 2020/03/24 · arxiv updated 2020/03/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Citations

• Networks of spiking neurons: The third generation of neural network models
• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Currents carried by sodium and potassium ions through the membrane of the giant axon of Loligo
• A 128×128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor
• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• Simple model of spiking neurons
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Mastering the game of Go with deep neural networks and tree search
• Fixed Point Quantization of Deep Convolutional Networks
• Neuromorphic Deep Learning Machines
• SLAYER: Spike Layer Error Reassignment in Time
• Backpropagation through time and the brain
• Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks
• Direct Training for Spiking Neural Networks: Faster, Larger, Better
• Learning Multiple Layers of Features from Tiny Images

Cited by

• A Study On the Effects of Pre-processing On Spatio-temporal Action Recognition Using Spiking Neural Networks Trained with STDP
• LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing