PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks

2024/06/29 by Markus Grotz, Grotz, Markus, Mohit Shridhar +5 · 14 citations
Engineering · Neuroscience · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Motor Control and Adaptation #Robot Manipulation and Learning #Robotics (cs.RO) #Teleoperation and Haptic Systems

paper · pdf · doi:10.48550/arxiv.2407.00278

openalex publication_date 2024/06/29 · openalex created_date 2024/07/03 · openalex updated_date 2026/07/28

Abstract

Bimanual manipulation is challenging due to precise spatial and temporal coordination required between two arms. While there exist several real-world bimanual systems, there is a lack of simulated benchmarks with a large task diversity for systematically studying bimanual capabilities across a wide range of tabletop tasks. This paper addresses the gap by extending RLBench to bimanual manipulation. We open-source our code and benchmark comprising 13 new tasks with 23 unique task variations, each requiring a high degree of coordination and adaptability. To kickstart the benchmark, we extended several state-of-the art methods to bimanual manipulation and also present a language-conditioned behavioral cloning agent -- PerAct2, which enables the learning and execution of bimanual 6-DoF manipulation tasks. Our novel network architecture efficiently integrates language processing with action prediction, allowing robots to understand and perform complex bimanual tasks in response to user-specified goals. Project website with code is available at: http://bimanual.github.io

Cited by

• Sampling-Based Optimization with Parallelized Physics Simulator for Bimanual Manipulation
• EL3DD: Extended Latent 3D Diffusion for Language Conditioned Multitask Manipulation
• TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models
• Embodiment Transfer Learning for Vision-Language-Action Models
• Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy
• Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
• VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation
• Large Pre-Trained Models for Bimanual Manipulation in 3D
• ROPA: Synthetic Robot Pose Generation for RGB-D Bimanual Data Augmentation
• PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
• GP3: A 3D Geometry-Aware Policy with Multi-View Images for Robotic Manipulation
• SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation
• Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
• Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation

Related

• Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework
• Benchmarking Generalizable Bimanual Manipulation: RoboTwin Dual-Arm Collaboration Challenge at CVPR 2025 MEIS Workshop
• Large Language Models for Orchestrating Bimanual Robots
• AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation
• Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers