New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles

2025/12/01 by Aref Ghoreishee, Abhishek Kumar Mishra, Ghoreishee, Aref +7
Computer Science · Engineering · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2512.01882

openalex publication_date 2025/12/01 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28

Abstract

This work proposes an end-to-end multi-modal reinforcement learning framework for high-level decision-making in autonomous vehicles. The framework integrates heterogeneous sensory input, including camera images, LiDAR point clouds, and vehicle heading information, through a cross-attention transformer-based perception module. Although transformers have become the backbone of modern multi-modal architectures, their high computational cost limits their deployment in resource-constrained edge environments. To overcome this challenge, we propose a spiking temporal-aware transformer-like architecture that uses ternary spiking neurons for computationally efficient multi-modal fusion. Comprehensive evaluations across multiple tasks in the Highway Environment demonstrate the effectiveness and efficiency of the proposed approach for real-time autonomous decision-making.

Citations

• Multimodal Representation Learning and Fusion
• Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
• Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer
• Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learning
• Evaluation of Encoding Schemes on Ubiquitous Sensor Signal for Spiking Neural Network
• Spiking Convolutional Neural Networks for Text Classification
• BLOS-BEV: Navigation Map Enhanced Lane Segmentation Network, Beyond Line of Sight
• High-Performance Temporal Reversible Spiking Neural Networks with O(L) Training Memory and O(1) Inference Cost
• Vision Language Models in Autonomous Driving: A Survey and Outlook
• RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection
• Spike-driven Transformer
• End-to-end Autonomous Driving: Challenges and Frontiers
• Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization
• Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL
• Reducing ANN-SNN Conversion Error through Residual Membrane Potential
• Spikformer: When Spiking Neural Network Meets Transformer
• Multi-modal Sensor Fusion for Auto Driving Perception: A Survey
• Optimized Potential Initialization for Low-latency Spiking Neural Networks
• Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation
• Training Spiking Neural Networks Using Lessons From Deep Learning
• Attention Bottlenecks for Multimodal Fusion
• Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning
• Deep Residual Learning in Spiking Neural Networks
• Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control
• Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by\n Implicitly Unprojecting to 3D
• Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
• End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• PointPillars: Fast Encoders for Object Detection from Point Clouds
• Going Deeper in Spiking Neural Networks: VGG and Residual Architectures
• Attention Is All You Need
• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
• Networks of spiking neurons: The third generation of neural network models

Related

• Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
• Autonomous Driving with Spiking Neural Networks
• SPformer: A Transformer Based DRL Decision Making Method for Connected Automated Vehicles
• M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving
• Parameterized Decision-making with Multi-modal Perception for Autonomous Driving