Revisit Self-supervised Depth Estimation with Local Structure-from-Motion

2024/07/27 by Zhu, Shengjie, Liu, Xiaoming
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2407.19166

Abstract

Both self-supervised depth estimation and Structure-from-Motion (SfM) recover scene depth from RGB videos. Despite sharing a similar objective, the two approaches are disconnected. Prior works of self-supervision backpropagate losses defined within immediate neighboring frames. Instead of learning-through-loss, this work proposes an alternative scheme by performing local SfM. First, with calibrated RGB or RGB-D images, we employ a depth and correspondence estimator to infer depthmaps and pair-wise correspondence maps. Then, a novel bundle-RANSAC-adjustment algorithm jointly optimizes camera poses and one depth adjustment for each depthmap. Finally, we fix camera poses and employ a NeRF, however, without a neural network, for dense triangulation and geometric verification. Poses, depth adjustments, and triangulated sparse depths are our outputs. For the first time, we show self-supervision within 5 frames already benefits SoTA supervised depth and correspondence models. The project page is held in the link (https://shngjz.github.io/SSfM.github.io/).

Related

• Self-Supervised Human Depth Estimation from Monocular Videos
• Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
• Self-Supervised Learning of Depth and Motion Under Photometric Inconsistency
• S³Net: Semantic-Aware Self-supervised Depth Estimation with Monocular Videos and Synthetic Data
• Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation