Temporal-wise Attention Spiking Neural Networks for Event Streams Classification

2021/07/25 by Man Yao, Huanhuan Gao, Yao, Man +12 · 13 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Neural Networks and Reservoir Computing #cs.CV

paper · pdf · doi:10.48550/arxiv.2107.11711

Accepted by ICCV 2021

arxiv created 2021/07/25 · openalex publication_date 2021/07/25 · arxiv updated 2021/07/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

How to effectively and efficiently deal with spatio-temporal event streams, where the events are generally sparse and non-uniform and have the microsecond temporal resolution, is of great value and has various real-life applications. Spiking neural network (SNN), as one of the brain-inspired event-triggered computing models, has the potential to extract effective spatio-temporal features from the event streams. However, when aggregating individual events into frames with a new higher temporal resolution, existing SNN models do not attach importance to that the serial frames have different signal-to-noise ratios since event streams are sparse and non-uniform. This situation interferes with the performance of existing SNNs. In this work, we propose a temporal-wise attention SNN (TA-SNN) model to learn frame-based representation for processing event streams. Concretely, we extend the attention concept to temporal-wise input to judge the significance of frames for the final decision at the training stage, and discard the irrelevant frames at the inference stage. We demonstrate that TA-SNN models improve the accuracy of event streams classification tasks. We also study the impact of multiple-scale temporal resolutions for frame-based representation. Our approach is tested on three different classification tasks: gesture recognition, image classification, and spoken digit recognition. We report the state-of-the-art results on these tasks, and get the essential improvement of accuracy (almost 19%) for gesture recognition with only 60 ms.

Cited by

• Efficient Spike-driven Transformer for High-performance Drone-View Geo-Localization
• Temporal Dynamics Enhancer for Directly Trained Spiking Object Detectors
• Parallel Delayed Memory Units for Enhanced Temporal Modeling in Biomedical and Bioacoustic Signal Analysis
• Large Language Models Inference Engines based on Spiking Neural Networks
• Event-based Facial Keypoint Alignment via Cross-Modal Fusion Attention and Self-Supervised Multi-Event Representation Learning
• EventTracer: Fast Path Tracing-based Event Stream Rendering
• Revealing Latent Information: A Physics-inspired Self-supervised Pre-training Framework for Noisy and Sparse Events
• Unleashing the Temporal Potential of Stereo Event Cameras for Continuous-Time 3D Object Detection
• Edge Intelligence with Spiking Neural Networks
• Fractional Spike Differential Equations Neural Network with Efficient Adjoint Parameters Training
• Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocation
• SPEAR: Structured Pruning for Spiking Neural Networks via Synaptic Operation Estimation and Reinforcement Learning
• Learning from Noise: Enhancing DNNs for Event-Based Vision through Controlled Noise Injection

Related

• Spiking Neural Networks for event-based action recognition: A new task to understand their advantage
• Razor SNN: Efficient Spiking Neural Network with Temporal Embeddings
• Ultra-low Latency Spiking Neural Networks with Spatio-Temporal Compression and Synaptic Convolutional Block
• A Spatial-channel-temporal-fused Attention for Spiking Neural Networks
• TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks