Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips

2024/02/15 by Man Yao, Jiakui Hu, Yao, Man +13 · 19 citations
Computer Science · Engineering · Neuroscience · #Advanced Memory and Neural Computing #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Neural dynamics and brain function

paper · pdf · doi:10.48550/arxiv.2404.03663

openalex publication_date 2024/02/15 · openalex created_date 2024/04/09 · openalex updated_date 2026/07/28

Abstract

Neuromorphic computing, which exploits Spiking Neural Networks (SNNs) on neuromorphic chips, is a promising energy-efficient alternative to traditional AI. CNN-based SNNs are the current mainstream of neuromorphic computing. By contrast, no neuromorphic chips are designed especially for Transformer-based SNNs, which have just emerged, and their performance is only on par with CNN-based SNNs, offering no distinct advantage. In this work, we propose a general Transformer-based SNN architecture, termed as ``Meta-SpikeFormer", whose goals are: 1) Lower-power, supports the spike-driven paradigm that there is only sparse addition in the network; 2) Versatility, handles various vision tasks; 3) High-performance, shows overwhelming performance advantages over CNN-based SNNs; 4) Meta-architecture, provides inspiration for future next-generation Transformer-based neuromorphic chip designs. Specifically, we extend the Spike-driven Transformer in \citetyao2023spike into a meta architecture, and explore the impact of structure, spike-driven self-attention, and skip connection on its performance. On ImageNet-1K, Meta-SpikeFormer achieves 80.0% top-1 accuracy (55M), surpassing the current state-of-the-art (SOTA) SNN baselines (66M) by 3.7%. This is the first direct training SNN backbone that can simultaneously supports classification, detection, and segmentation, obtaining SOTA results in SNNs. Finally, we discuss the inspiration of the meta SNN architecture for neuromorphic chip design. Source code and models are available at \urlhttps://github.com/BICLab/Spike-Driven-Transformer-V2.

Cited by

• Signal-SGN++: Topology-Enhanced Time-Frequency Spiking Graph Network for Skeleton-Based Action Recognition
• Surgical Scene Segmentation using a Spike-Driven Video Transformer with Real-Time Potential
• Efficient Spike-driven Transformer for High-performance Drone-View Geo-Localization
• Temporal Dynamics Enhancer for Directly Trained Spiking Object Detectors
• Learning Scalable Temporal Representations in Spiking Neural Networks Without Labels
• Attention via Synaptic Plasticity is All You Need: A Biologically Inspired Spiking Neuromorphic Transformer
• I2E: Real-Time Image-to-Event Conversion for High-Performance Spiking Neural Networks
• SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognition
• Unveiling the Spatial-temporal Effective Receptive Fields of Spiking Neural Networks
• Decoding Listeners Identity: Person Identification from EEG Signals Using a Lightweight Spiking Transformer
• SpikePool: Event-driven Spiking Transformer with Pooling Attention
• Efficient Training of Spiking Neural Networks by Spike-aware Data Pruning
• Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation
• SpikingBrain: Spiking Brain-inspired Large Models
• SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos
• SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration
• Learning Internal Biological Neuron Parameters and Complexity-Based Encoding for Improved Spiking Neural Networks Performance
• Toward Efficient Spiking Transformers: Synapse Pruning Meets Synergistic Learning-Based Compensation
• STF: Shallow-Level Temporal Feedback to Enhance Spiking Transformers

Related

• Spike-driven Transformer
• Spikingformer: A Key Foundation Model for Spiking Neural Networks
• Spikformer: When Spiking Neural Network Meets Transformer
• MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion
• Spiking Neural Networks: The Future of Brain-Inspired Computing