Single RGB-D Camera Teleoperation for General Robotic Manipulation

2021/06/28 by Quan Vuong, Yuzhe Qin, Vuong, Quan +9
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Augmented Reality Applications #FOS: Computer and information sciences #Robotics (cs.RO) #Robotics and Sensor-Based Localization #Teleoperation and Haptic Systems

paper · pdf · doi:10.48550/arxiv.2106.14396

openalex publication_date 2021/06/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a teleoperation system that uses a single RGB-D camera as the human motion capture device. Our system can perform general manipulation tasks such as cloth folding, hammering and 3mm clearance peg in hole. We propose the use of non-Cartesian oblique coordinate frame, dynamic motion scaling and reposition of operator frames to increase the flexibility of our teleoperation system. We hypothesize that lowering the barrier of entry to teleoperation will allow for wider deployment of supervised autonomy system, which will in turn generates realistic datasets that unlock the potential of machine learning for robotic manipulation. Demo of our systems are available online https://sites.google.com/view/manipulation-teleop-with-rgbd

Citations

• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data\n for Imitation
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation
• DoorGym: A Scalable Door Opening Environment And Baseline Agent
• A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU
• Language Models are Few-Shot Learners
• Visual Imitation Made Easy
• FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
• Human-in-the-Loop Imitation Learning using Remote Teleoperation
• Auto-Tuned Sim-to-Real Transfer
• Actionable Models: Unsupervised Offline Reinforcement Learning of\n Robotic Skills

Related

• Semi-Autonomous Teleoperation via Learning Non-Prehensile Manipulation Skills
• From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation
• DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
• TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation
• RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation