Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

2025/05/20 by Jiangrong Shen, Shen, Jiangrong, Yulin Xie +9 · 1 citation
Computer Science · Neuroscience · #EEG and Brain-Computer Interfaces #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Music and Audio Processing #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2505.14535

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55%, 70.65% and 97.5%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.

Citations

• Full-DoF Egomotion Estimation for Event Cameras Using Geometric Solvers
• Differential Coding for Training-Free ANN-to-SNN Conversion
• Accurate Pose Estimation for Flight Platforms based on Divergent Multi-Aperture Imaging System
• 3D Trajectory Reconstruction of Moving Points Based on a Monocular Camera
• High-precision visual navigation device calibration method based on collimator
• SNN-Driven Multimodal Human Action Recognition via Sparse Spatial-Temporal Data Fusion
• Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency
• Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
• Ridge Estimation-Based Vision and Laser Ranging Fusion Localization Method for UAVs
• Optical Flow-Guided 6DoF Object Pose Tracking with an Event Camera
• Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model
• Digit Recognition using Multimodal Spiking Neural Networks
• Line-based 6-DoF Object Pose Estimation and Tracking With an Event Camera
• Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis
• LECalib: Line-Based Event Camera Calibration
• MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition
• Enhancing Adaptive History Reserving by Spiking Convolutional Block Attention Module in Recurrent Neural Networks
• Darwin3: A large-scale neuromorphic chip with a Novel ISA and On-Chip Learning
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
• SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
• Boosting Multi-modal Model Performance with Adaptive Gradient Modulation
• SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition
• ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks
• PMR: Prototypical Modal Rebalance for Multimodal Learning
• Balanced Multimodal Learning via On-the-fly Gradient Modulation
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks
• What Makes Multi-modal Learning Better than Single (Provably)
• What Makes Training Multi-Modal Classification Networks Hard?
• Audio-Visual Event Localization in Unconstrained Videos
• Networks of spiking neurons: The third generation of neural network models

Cited by

• SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition

Related

• Enhancing SNN-based Spatio-Temporal Learning: A Benchmark Dataset and Cross-Modality Attention Model
• Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
• Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence
• Modality-Dependent Memory Mechanisms in Cross-Modal Neuromorphic Computing
• Multi-Plasticity Synergy with Adaptive Mechanism Assignment for Training Spiking Neural Networks