Dynamic Handover: Throw and Catch with Bimanual Hands

2023/09/11 by Binghao Huang, Yuanpei Chen, Huang, Binghao +11 · 9 citations
Computer Science · Engineering · #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2309.05655

openalex publication_date 2023/09/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Humans throw and catch objects all the time. However, such a seemingly common skill introduces a lot of challenges for robots to achieve: The robots need to operate such dynamic actions at high-speed, collaborate precisely, and interact with diverse objects. In this paper, we design a system with two multi-finger hands attached to robot arms to solve this problem. We train our system using Multi-Agent Reinforcement Learning in simulation and perform Sim2Real transfer to deploy on the real robots. To overcome the Sim2Real gap, we provide multiple novel algorithm designs including learning a trajectory prediction model for the object. Such a model can help the robot catcher has a real-time estimation of where the object will be heading, and then react accordingly. We conduct our experiments with multiple objects in the real-world system, and show significant improvements over multiple baselines. Our project page is available at \urlhttps://binghao-huang.github.io/dynamic_handover/.

Cited by

• Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
• Learning Dexterous Manipulation Skills from Imperfect Simulations
• GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies
• VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning
• Leveraging Neural Descriptor Fields for Learning Contact-Aware Dynamic Recovery
• DA-MMP: Learning Coordinated and Accurate Throwing with Dynamics-Aware Motion Manifold Primitives
• DyDexHandover: Human-like Bimanual Dynamic Dexterous Handover using RGB-only Perception
• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
• DexH2R: A Benchmark for Dynamic Dexterous Grasping in Human-to-Robot Handover

Related

• Flexible Handover with Real-Time Robust Dynamic Grasp Trajectory Generation
• DyDexHandover: Human-like Bimanual Dynamic Dexterous Handover using RGB-only Perception
• Learning Dynamic Robot-to-Human Object Handover from Human Feedback
• Modeling Dynamic Hand-Object Interactions with Applications to Human-Robot Handovers
• Learning-based Dynamic Robot-to-Human Handover