Dual-Stream Attention with Multi-Modal Queries for Object Detection in Transportation Applications

2025/08/06 by Anwar, Noreen, Bilodeau, Guillaume-Alexandre, Bouachir, Wassim
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.04868

Abstract

Transformer-based object detectors often struggle with occlusions, fine-grained localization, and computational inefficiency caused by fixed queries and dense attention. We propose DAMM, Dual-stream Attention with Multi-Modal queries, a novel framework introducing both query adaptation and structured cross-attention for improved accuracy and efficiency. DAMM capitalizes on three types of queries: appearance-based queries from vision-language models, positional queries using polygonal embeddings, and random learned queries for general scene coverage. Furthermore, a dual-stream cross-attention module separately refines semantic and spatial features, boosting localization precision in cluttered scenes. We evaluated DAMM on four challenging benchmarks, and it achieved state-of-the-art performance in average precision (AP) and recall, demonstrating the effectiveness of multi-modal query adaptation and dual-stream attention. Source code is at: \hrefhttps://github.com/DET-LIP/DAMMGitHub.

Citations

• UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery
• Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
• Micrograph segmentations for DDEVD
• Segment Anything
• Cascaded Zoom-in Detector for High Resolution Aerial Images
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
• DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
• DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
• DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
• Grounded Language-Image Pre-training
• Conditional DETR for Fast Training Convergence
• QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection
• Learning Transferable Visual Models From Natural Language Supervision
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Poly-YOLO: higher speed, more precise detection and instance segmentation for YOLOv3
• End-to-End Object Detection with Transformers
• Detection and Tracking Meet Drones Challenge
• PolarMask: Single Shot Instance Segmentation with Polar Representation
• Clustered Object Detection in Aerial Images
• Generalized Intersection over Union: A Metric and A Loss for Bounding\n Box Regression
• The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking

Related

• Guiding Query Position and Performing Similar Attention for Transformer-Based Detection Heads
• Dual-R-DETR: Resolving Query Competition with Pairwise Routing in Transformer Decoders
• Multi-modal Queried Object Detection in the Wild
• DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention
• DQ3D: Depth-guided Query for Transformer-Based 3D Object Detection in Traffic Scenarios