PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation

2017/11/29 by Danfei Xu, Dragomir Anguelov, Xu, Danfei +3 · 9 citations
Earth and Planetary Sciences · Engineering · Environmental Science · #3D Surveying and Cultural Heritage #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Remote Sensing and LiDAR Applications #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.1711.10871

openalex publication_date 2017/11/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information. Unlike existing methods that either use multi-stage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic. The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively. The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors. We evaluate PointFusion on two distinctive datasets: the KITTI dataset that features driving scenes captured with a lidar-camera setup, and the SUN-RGBD dataset that captures indoor environments with RGB-D cameras. Our model is the first one that is able to perform better or on-par with the state-of-the-art on these diverse datasets without any dataset-specific model tuning.

Citations

• Feature Pyramid Networks for Object Detection
• Deep Residual Learning for Image Recognition

Cited by

• SimBEV2X: A Large-Scale Dataset and Data Generation Tool for Multi-Task Vehicle-to-Everything Cooperative Perception
• Class-specific Anchoring Proposal for 3D Object Recognition in LIDAR and RGB Images
• 3D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinement
• Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge
• Open-Vocabulary Spatio-Temporal Scene Graph for Robot Perception and Teleoperation Planning
• View Invariant Human Body Detection and Pose Estimation from Multiple Depth Sensors
• Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
• Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
• Object Recognition Datasets and Challenges: A Review

Related

• RoIFusion: 3D Object Detection from LiDAR and Vision
• Frustum PointNets for 3D Object Detection from RGB-D Data
• BOX3D: Lightweight Camera-LiDAR Fusion for 3D Object Detection and Localization
• From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network
• Multi-View 3D Object Detection Network for Autonomous Driving