SfM-Net: Learning of Structure and Motion from Video

2017/04/25 by Sudheendra Vijayanarasimhan, Susanna Ricco, Vijayanarasimhan, Sudheendra +7 · 29 citations
Computer Science · Engineering · #Advanced Vision and Imaging #Robotics and Sensor-Based Localization #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.1704.07804

Abstract

We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.

Citations

• Unsupervised Monocular Depth Estimation with Left-Right Consistency

Cited by

• A Minimal Solver for Relative Pose Estimation with Unknown Focal Length from Two Affine Correspondences
• Globally Optimal Solution to the Generalized Relative Pose Estimation Problem using Affine Correspondences
• Self-Supervised Learning of Depth and Ego-motion with Differentiable Bundle Adjustment
• Rethinking the Faster R-CNN Architecture for Temporal Action Localization
• Image Matching Across Wide Baselines: From Paper to Practice
• Toward Hierarchical Self-Supervised Monocular Absolute Depth Estimation for Autonomous Driving Applications
• Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments
• Consistent Video Depth Estimation
• VisualEchoes: Spatial Image Representation Learning through Echolocation
• InverseRenderNet: Learning single image inverse rendering
• Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss
• RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training
• Scene-Centric Unsupervised Video Panoptic Segmentation
• Layer-structured 3D Scene Inference via View Synthesis
• Robust Consistent Video Depth Estimation
• TransCamP: Graph Transformer for 6-DoF Camera Pose Estimation
• Unsupervised part representation by Flow Capsules
• A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
• Multiview Cross-supervision for Semantic Segmentation
• Self-Supervised Learning of Depth and Camera Motion from 360° Videos
• Object category learning and retrieval with weak supervision
• Learning Correspondence from the Cycle-Consistency of Time
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• Movement science needs different pose tracking algorithms
• Semantic-Guided Representation Enhancement for Self-supervised Monocular Trained Depth Estimation
• Towards Real-Time Monocular Depth Estimation for Robotics: A Survey
• Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
• Indoor GeoNet: Weakly Supervised Hybrid Learning for Depth and Pose Estimation
• Beyond the Camera: Neural Networks in World Coordinates

Related

• Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding
• Learning to Extract Motion from Videos in Convolutional Neural Networks
• ENG: End-to-end Neural Geometry for Robust Depth and Pose Estimation using CNNs
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos
• Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding