Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360° Panoramic Imagery

2018/08/19 by Grégoire Payen de La Garanderie, de La Garanderie, Grégoire Payen, Amir Atapour–Abarghouei +3 · 1 citation
Computer Science · Engineering · #Advanced Neural Network Applications #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.1808.06253

openalex publication_date 2018/08/19 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

Recent automotive vision work has focused almost exclusively on processing forward-facing cameras. However, future autonomous vehicles will not be viable without a more comprehensive surround sensing, akin to a human driver, as can be provided by 360° panoramic cameras. We present an approach to adapt contemporary deep network architectures developed on conventional rectilinear imagery to work on equirectangular 360° panoramic imagery. To address the lack of annotated panoramic automotive datasets availability, we adapt a contemporary automotive dataset, via style and projection transformations, to facilitate the cross-domain retraining of contemporary algorithms for panoramic imagery. Following this approach we retrain and adapt existing architectures to recover scene depth and 3D pose of vehicles from monocular panoramic imagery without any panoramic training labels or calibration parameters. Our approach is evaluated qualitatively on crowd-sourced panoramic images and quantitatively using an automotive environment simulator to provide the first benchmark for such techniques within panoramic imagery.

Citations

• Vision meets robotics: The KITTI dataset
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Adversarial Feature Learning
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry\n and Semantics
• Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching
• Adversarial Discriminative Domain Adaptation
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Accurate Single Stage Detector Using Recurrent Rolling Convolution

Cited by

• PanoSplatt3R: Leveraging Perspective Pretraining for Generalized Unposed Wide-Baseline Panorama Reconstruction

Related

• Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving
• OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving
• Calibrating Panoramic Depth Estimation for Practical Localization and Mapping
• MonoPP: Metric-Scaled Self-Supervised Monocular Depth Estimation by Planar-Parallax Geometry in Automotive Applications
• A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View