2020/06/23 by Aljaž Božič, Božič, Aljaž, Pablo Palafox +9 · 3 citations
Computer Science · Earth and Planetary Sciences · #3D Surveying and Cultural Heritage #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Graphics (cs.GR) #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Optical measurement and interference techniques #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2006.13240
openalex publication_date 2020/06/23 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization. Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences and their confidences. These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem. By enabling gradient back-propagation through the weighted non-linear least squares solver, we are able to learn correspondences and confidences in an end-to-end manner such that they are optimal for the task of non-rigid tracking. Under this formulation, correspondence confidences can be learned via self-supervision, informing a learned robust optimization, where outliers and wrong correspondences are automatically down-weighted to enable effective tracking. Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85 times faster correspondence prediction than comparable deep-learning based methods. We make our code available.