2020/08/16 by Ming Zhu, Zhu, Ming, Chao Ma +5 · 2 citations
Computer Science · Earth and Planetary Sciences · Engineering · #3D Surveying and Cultural Heritage #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics and Sensor-Based Localization #cs.CV
paper · pdf · doi:10.48550/arxiv.2008.10436
Accepted by WACV 2021
arxiv created 2020/08/16 · openalex publication_date 2020/08/16 · arxiv updated 2020/08/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper, we focus on exploring the fusion of images and point clouds for 3D object detection in view of the complementary nature of the two modalities, i.e., images possess more semantic information while point clouds specialize in distance sensing. To this end, we present a novel two-stage multi-modal fusion network for 3D object detection, taking both binocular images and raw point clouds as input. The whole architecture facilitates two-stage fusion. The first stage aims at producing 3D proposals through sparse point-wise feature fusion. Within the first stage, we further exploit a joint anchor mechanism that enables the network to utilize 2D-3D classification and regression simultaneously for better proposal generation. The second stage works on the 2D and 3D proposal regions and fuses their dense features. In addition, we propose to use pseudo LiDAR points from stereo matching as a data augmentation method to densify the LiDAR points, as we observe that objects missed by the detection network mostly have too few points especially for far-away objects. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.