2017/10/02 by Arunkumar Byravan, Byravan, Arunkumar, Felix Leeb +5 · 4 citations
Computer Science · Engineering · #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Human Motion and Animation #Human Pose and Action Recognition #Neural and Evolutionary Computing (cs.NE) #Robot Manipulation and Learning #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.1710.00489
openalex publication_date 2017/10/02 · openalex created_date 2022/08/29 · openalex updated_date 2026/07/28
In this work, we present an approach to deep visuomotor control using\nstructured deep dynamics models. Our deep dynamics model, a variant of\nSE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an\nencoder-decoder structure. Unlike prior work, our dynamics model is structured:\ngiven an input scene, our network explicitly learns to segment salient parts\nand predict their pose-embedding along with their motion modeled as a change in\nthe pose space due to the applied actions. We train our model using a pair of\npoint clouds separated by an action and show that given supervision only in the\nform of point-wise data associations between the frames our network is able to\nlearn a meaningful segmentation of the scene along with consistent poses. We\nfurther show that our model can be used for closed-loop control directly in the\nlearned low-dimensional pose space, where the actions are computed by\nminimizing error in the pose space using gradient-based methods, similar to\ntraditional model-based control. We present results on controlling a Baxter\nrobot from raw depth data in simulation and in the real world and compare\nagainst two baseline deep networks. Our method runs in real-time, achieves good\nprediction of scene dynamics and outperforms the baseline methods on multiple\ncontrol runs. Video results can be found at:\nhttps://rse-lab.cs.washington.edu/se3-structured-deep-ctrl/\n