The fine line between dead neurons and sparsity in binarized spiking neural networks

2022/01/28 by Jason K. Eshraghian, Wei Lü, Eshraghian, Jason K. +1 · 1 citation
Computer Science · Engineering · Neuroscience · #Advanced Memory and Neural Computing #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Neural dynamics and brain function #Neurons and Cognition (q-bio.NC)

paper · pdf · doi:10.48550/arxiv.2201.11915

openalex publication_date 2022/01/28 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28

Abstract

Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/

Cited by

• SpikeFit: Towards Optimal Deployment of Spiking Networks on Neuromorphic Hardware

Related

• Spiking Neural Networks: a theoretical framework for Universal Approximation and training
• To Spike or Not to Spike, that is the Question
• Explicitly Trained Spiking Sparsity in Spiking Neural Networks with Backpropagation
• A Synapse-Threshold Synergistic Learning Approach for Spiking Neural Networks
• Inherent Redundancy in Spiking Neural Networks