Deep Imitation Learning for Bimanual Robotic Manipulation

2020/10/11 by Fan Xie, Alexander Chowdhury, Xie, Fan +9 · 9 citations
Engineering · Computer Science · #Robot Manipulation and Learning #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2010.05134

Abstract

We present a deep imitation learning framework for robotic bimanual manipulation in a continuous state-action space. A core challenge is to generalize the manipulation skills to objects in different locations. We hypothesize that modeling the relational information in the environment can significantly improve generalization. To achieve this, we propose to (i) decompose the multi-modal dynamics into elemental movement primitives, (ii) parameterize each primitive using a recurrent graph neural network to capture interactions, and (iii) integrate a high-level planner that composes primitives sequentially and a low-level controller to combine primitive dynamics and inverse kinematics control. Our model is a deep, hierarchical, modular architecture. Compared to baselines, our model generalizes better and achieves higher success rates on several simulated bimanual robotic manipulation tasks. We open source the code for simulation, data, and models at: https://github.com/Rose-STL-Lab/HDR-IL.

Citations

• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Sequence to Sequence Learning with Neural Networks
• Dual arm manipulation—A survey

Cited by

• Masked Generative Policy for Robotic Control
• GBAGC-RL: Goal-based arm-gripper coordination reinforcement learning approach for robotic manipulation skills
• RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation
• TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models
• VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation
• ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation
• SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation
• Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control
• CollaBot: Vision-Language Guided Simultaneous Collaborative Manipulation

Related

• Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Models
• A Comparison of Imitation Learning Algorithms for Bimanual Manipulation
• BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation
• Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers
• A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies