Cross Modal Transformer: Towards Fast and Robust 3D Object Detection

2023/01/03 by Yan, Junjie, Liu, Yingfei, Sun, Jianjian +4 · 16 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2301.01283

Abstract

In this paper, we propose a robust 3D detector, named Cross Modal Transformer (CMT), for end-to-end 3D multi-modal detection. Without explicit view transformation, CMT takes the image and point clouds tokens as inputs and directly outputs accurate 3D bounding boxes. The spatial alignment of multi-modal tokens is performed by encoding the 3D points into multi-modal features. The core design of CMT is quite simple while its performance is impressive. It achieves 74.1% NDS (state-of-the-art with single model) on nuScenes test set while maintaining fast inference speed. Moreover, CMT has a strong robustness even if the LiDAR is missing. Code is released at https://github.com/junjie18/CMT.

Cited by

• End-to-End 3-D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration
• UP-Fuse: Uncertainty-guided LiDAR-Camera Fusion for 3D Panoptic Segmentation
• LiteFusion: Taming 3D Object Detectors from Vision-Based to Multi-Modal with Minimal Adaptation
• ALIGN: Advanced Query Initialization with LiDAR-Image Guidance for Occlusion-Robust 3D Object Detection
• Cross-Level Sensor Fusion with Object Lists via Transformer for 3D Object Detection
• FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detection
• UniLION: Towards Unified Autonomous Driving Model with Linear Group RNNs
• Multi-Sensor Alignment for Weather Simulations
• An Analytical Framework to Enhance Autonomous Vehicle Perception for Smart Cities
• CrossRay3D: Geometry and Distribution Guidance for Efficient Multimodal 3D Detection
• Towards Neurocognitive-Inspired Intelligence: From AI's Structural Mimicry to Human-Like Functional Cognition
• CRAB: Camera-Radar Fusion for Reducing Depth Ambiguity in Backward Projection based View Transformation
• SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather
• CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction
• Recent Advances in Transformer and Large Language Models for UAV Applications
• Collaborative Perceiver: Elevating Vision-based 3D Object Detection via Local Density-Aware Spatial Occupancy

Related

• Unifying Voxel-based Representation with Transformer for 3D Object Detection
• CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection
• Multi-Modal 3D Object Detection by Box Matching
• CrossFusion: Interleaving Cross-modal Complementation for Noise-resistant 3D Object Detection
• Robust Multimodal 3D Object Detection via Modality-Agnostic Decoding and Proximity-based Modality Ensemble