RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

2022/10/21 by Tang, Zhenggang, Sundaralingam, Balakumar, Tremblay, Jonathan +6 · 3 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2210.11668

Abstract

We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.

Cited by

• ReMoSPLAT: Reactive Mobile Manipulation Control on a Gaussian Splat
• TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
• RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

Related

• RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation
• RGBGrasp: Image-based Object Grasping by Capturing Multiple Views during Robot Arm Movement with Neural Radiance Fields
• Object Modelling with a Handheld RGB-D Camera
• SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
• Multi-View Fusion for Multi-Level Robotic Scene Understanding