IML-Spikeformer: Input-aware Multi-Level Spiking Transformer for Speech Processing

2025/07/10 by Zuojian Song, Shimin Zhang, Song, Zeyang +7
Computer Science · #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Multimedia (cs.MM) #Sound (cs.SD) #Speech Recognition and Synthesis #Speech and Audio Processing #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2507.07396

openalex publication_date 2025/07/10 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Spiking Neural Networks (SNNs), inspired by biological neural mechanisms, represent a promising neuromorphic computing paradigm that offers energy-efficient alternatives to traditional Artificial Neural Networks (ANNs). Despite proven effectiveness, SNN architectures have struggled to achieve competitive performance on large-scale speech processing tasks. Two key challenges hinder progress: (1) the high computational overhead during training caused by multi-timestep spike firing, and (2) the absence of large-scale SNN architectures tailored to speech processing tasks. To overcome the issues, we introduce Input-aware Multi-Level Spikeformer, i.e. IML-Spikeformer, a spiking Transformer architecture specifically designed for large-scale speech processing. Central to our design is the Input-aware Multi-Level Spike (IMLS) mechanism, which simulates multi-timestep spike firing within a single timestep using an adaptive, input-aware thresholding scheme. IML-Spikeformer further integrates a Re-parameterized Spiking Self-Attention (RepSSA) module with a Hierarchical Decay Mask (HDM), forming the HD-RepSSA module. This module enhances the precision of attention maps and enables modeling of multi-scale temporal dependencies in speech signals. Experiments demonstrate that IML-Spikeformer achieves word error rates of 6.0% on AiShell-1 and 3.4% on Librispeech-960, comparable to conventional ANN transformers while reducing theoretical inference energy consumption by 4.64× and 4.32× respectively. IML-Spikeformer marks an advance of scalable SNN architectures for large-scale speech processing in both task performance and energy efficiency. Our source code and model checkpoints are publicly available at github.com/Pooookeman/IML-Spikeformer.

Citations

• Scaling Spike-Driven Transformer With Efficient Spike Firing Approximation Training
• Towards Ultra-Low-Power Neuromorphic Speech Enhancement with Spiking-FullSubNet
• Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model
• DPSNN: Spiking Neural Network for Low-Latency Streaming Speech Enhancement
• ED-sKWS: Early-Decision Spiking Neural Networks for Rapid,and Energy-Efficient Keyword Spotting
• SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
• Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips
• Learning Long Sequences in Spiking Neural Networks
• SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
• Efficient Video and Audio processing with Loihi 2
• Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
• Spiking Structured State Space Model for Monaural Speech Enhancement
• TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling
• SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation
• Spike-driven Transformer
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks
• NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems
• Transformers in Speech Processing: A Survey
• The Intel Neuromorphic DNS Challenge
• SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
• Complex Dynamic Neurons Improved Spiking Transformer Network for Efficient Automatic Speech Recognition
• Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition
• The Devil in Linear Transformer
• Understanding the Failure of Batch Normalization for Transformers in NLP
• Training Spiking Neural Networks with Local Tandem Learning
• Online Training Through Time for Spiking Neural Networks
• Masked Spiking Transformer
• Spikformer: When Spiking Neural Network Meets Transformer
• Attention Spiking Neural Networks
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation
• Efficient and Accurate Conversion of Spiking Neural Network with Burst Spikes
• MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
• Advancing Spiking Neural Networks towards Deep Residual Learning
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• Training Spiking Neural Networks Using Lessons From Deep Learning
• Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning
• A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware
• SpeechBrain: A General-Purpose Speech Toolkit
• Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks
• Deep Residual Learning in Spiking Neural Networks
• Low-activity supervised convolutional spiking neural networks applied to speech commands recognition
• Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
• Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
• An efficient and perceptually motivated auditory neural encoding and decoding algorithm for spiking neural networks
• Neural Population Coding for Effective Temporal Classification
• Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks
• ESPnet: End-to-End Speech Processing Toolkit
• Long short-term memory and learning-to-learn in networks of spiking neurons
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline
• Attention Is All You Need
• SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation
• Networks of spiking neurons: The third generation of neural network models

Related

• Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
• SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network
• SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
• SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognition
• STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking