RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
2023/07/02 by Haoshu Fang, Hongjie Fang, Fang, Hao-Shu +12 · 104 citations
Engineering · Computer Science · #Robot Manipulation and Learning #Multimodal Machine Learning Applications #Human Pose and Action Recognition
paper · pdf · doi:10.48550/arxiv.2307.00595
Abstract
A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations. This feature is attractive for enabling robots to acquire new skills and improving task and motion planning. However, due to limitations in the training dataset, the current focus of the community has mainly been on simple cases, such as push or pick-place tasks, relying solely on visual guidance. In reality, there are many complex skills, some of which may even require both visual and tactile perception to solve. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception. To achieve this, we have collected a dataset comprising over 110,000 contact-rich robot manipulation sequences across diverse skills, contexts, robots, and camera viewpoints, all collected in the real world. Each sequence in the dataset includes visual, force, audio, and action information. Moreover, we also provide a corresponding human demonstration video and a language description for each robot sequence. We have invested significant efforts in calibrating all the sensors and ensuring a high-quality dataset. The dataset is made publicly available at rh20t.github.io
Cited by
- Emergence of Human to Robot Transfer in Vision-Language-Action Models
- Being-H0.7: A Latent World-Action Model from Egocentric Videos
- StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision
- Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting
- OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning
- Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy
- An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges
- MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence
- Invariance Co-training for Robot Visual Generalization
- Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation
- IGen: Scalable Data Generation for Robot Learning from Open-World Images
- RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation
- FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models
- Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective
- DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models
- Co-Evolving Latent Action World Models
- HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
- One-shot Humanoid Whole-body Motion Learning
- A Humanoid Visual-Tactile-Action Dataset for Contact-Rich Manipulation
- RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation
- Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos
- Towards Automated Chicken Deboning via Learning-based Dynamically-Adaptive 6-DoF Multi-Material Cutting
- Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
- ARMADA: Autonomous Online Failure Detection and Human Shared Control Empower Scalable Real-world Deployment and Adaptation
- Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation
- Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
- Do You Know Where Your Camera Is? View-Invariant Policy Learning with Camera Conditioning
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models
- AIRoA MoMa Dataset: A Large-Scale Hierarchical Dataset for Mobile Manipulation
- VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation
- Generalist Robot Manipulation beyond Action Labeled Data
- FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models
- What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations
- Pure Vision Language Action (VLA) Models: A Comprehensive Survey
- Learning Dexterous Manipulation with Quantized Hand State
- History-Aware Visuomotor Policy Learning via Point Tracking
- A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
- Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
- OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
- Igniting VLMs toward the Embodied Space
- Multi-View 3D Point Tracking
- Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
- Survey of Vision-Language-Action Models for Embodied Manipulation
- Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policy
- Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
- 3D FlowMatch Actor: Unified 3D Policy for Single- and Dual-Arm Manipulation
- ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceiver
- Leveraging OS-Level Primitives for Robotic Action Management
- MolmoAct: Action Reasoning Models that can Reason in Space
- Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
- BEVCon: Advancing Bird's Eye View Perception with Contrastive Learning
- villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
- H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation
- Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding
- VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback
- Is Diversity All You Need for Scalable Robotic Manipulation?
- VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting
- TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
- HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning
- RGC-VQA: An Exploration Database for Robotic-Generated Video Quality Assessment
- Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
- DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policy
- Real-Time Execution of Action Chunking Flow Policies
- RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
- RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
- CEED-VLA: Consistency Vision-Language-Action Model with Early-Exit Decoding
- What Matters in Learning from Large-Scale Datasets for Robot Manipulation
- A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
- Human-assisted Robotic Policy Refinement via Action Preference Optimization
- Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondence
- 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
- Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning
- DeepVerse: 4D Autoregressive Video Generation as a World Model
- Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better
- Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptation
- ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation
- What Do Latent Action Models Actually Learn?
- Learning a Unified Policy for Position and Force Control in Legged Loco-Manipulation
- Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
- ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
- ScanBot: Towards Intelligent Surface Scanning in Embodied Robotic Systems
- TWINS: A Tactile Wearable Isomorphic Arm Networked System for Contact-Rich Manipulation Learning
- CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning
- BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization
- A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics
- DynamicManip: Enabling Dynamic Manipulation from a Single Static Demonstration
- H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
- EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation
- DataMIL: Selecting Data for Robot Imitation Learning with Datamodels
- Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware
- Augmented Reality for RObots (ARRO): Pointing Visuomotor Policies Towards Visual Robustness
- Towards Embodiment Scaling Laws in Robot Locomotion
- SIME: Enhancing Policy Self-Improvement with Modal-level Exploration
- Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
- Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
- Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments
- OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
- RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning
- SSC: A Verifiable Structured Representation for Bimanual Manipulation Labelling
- π0.5: a Vision-Language-Action Model with Open-World Generalization
- Dexterous Manipulation through Imitation Learning: A Survey
- Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning
Related