SAPIEN: A SimulAted Part-based Interactive ENvironment
2020/03/19 by Xiang, Fanbo, Qin, Yuzhe, Mo, Kaichun +11 · 99 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Robotics (cs.RO)
paper · doi:10.48550/arxiv.2003.08515
Abstract
Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.
Cited by
- The Curse of Precision: A Data Scaling Law for High-Precision Robotic Manipulation
- Feedforward 3D Editing Learns from Semantic-Part Transformation
- TongSIM: A General Platform for Simulating Intelligent Machines
- DeliveryBench: Can Agents Earn Profit in Real World?
- AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learning
- ART: Articulated Reconstruction Transformer
- Particulate: Feed-Forward 3D Object Articulation
- DragMesh: Interactive 3D Generation Made Easy
- Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
- Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation
- Embodied Co-Design for Rapidly Evolving Agents: Taxonomy, Frontiers, and Challenges
- GAOT: Generating Articulated Objects Through Text-Guided Diffusion Models
- YOLOA: Real-Time Affordance Detection via LLM Adapter
- PerFACT: Motion Policy with LLM-Powered Dataset Synthesis and Fusion Action-Chunking Transformers
- SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge
- TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
- RealAppliance: Let High-fidelity Appliance Assets Controllable and Workable as Aligned Real Manuals
- Language-guided 3D scene synthesis for fine-grained functionality understanding
- UniArt: Unified 3D Representation for Generating 3D Articulated Objects with Open-Set Articulation
- Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
- Wanderland: Geometrically Grounded Simulation for Open-World Embodied AI
- Learning to Generate Human-Human-Object Interactions from Textual Descriptions
- ArtiBench and ArtiBrain: Benchmarking Generalizable Vision-Language Articulated Object Manipulation
- GigaWorld-0: World Models as Data Engine to Empower Embodied AI
- ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes
- ArticFlow: Generative Simulation of Articulated Mechanisms
- SPAGS: Sparse-View Articulated Object Reconstruction from Single State via Planar Gaussian Splatting
- REArtGS++: Generalizable Articulation Reconstruction with Temporal Geometry Constraint via Planar Gaussian Splatting
- PhysX-Anything: Simulation-Ready Physical 3D Assets from Single Image
- LARM: A Large Articulated-Object Reconstruction Model
- Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects
- ArtReg: Visuo-Tactile based Pose Tracking and Manipulation of Unseen Articulated Objects
- Exploring Category-level Articulated Object Pose Tracking on SE(3) Manifolds
- Canonical Space Representation for 4D Panoptic Segmentation of Articulated Objects
- Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions
- LEGO-Eval: Towards Fine-Grained Evaluation on Synthesizing 3D Embodied Environments with Tool Augmentation
- FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion
- Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Models
- Nautilus: From One Prompt to Plug-and-Play Robot Learning
- SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representation
- NEBULA: Do We Evaluate Vision-Language-Action Agents Correctly?
- Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications
- PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding
- GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation
- Advances in 4D Representation: Geometry, Motion, and Interaction
- Towards Automated Chicken Deboning via Learning-based Dynamically-Adaptive 6-DoF Multi-Material Cutting
- RAPID Hand Prototype: Design of an Affordable, Fully-Actuated Biomimetic Hand for Dexterous Teleoperation
- Kinematic Kitbashing for Modeling Functional Articulated Objects
- EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making
- DemoHLM: From One Demonstration to Generalizable Humanoid Loco-Manipulation
- Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
- Vi-TacMan: Articulated Object Manipulation via Vision and Touch
- MobRT: A Digital Twin-Based Framework for Scalable Learning in Mobile Manipulation
- OceanGym: A Benchmark Environment for Underwater Embodied Agents
- SRMP: Search-Based Robot Motion Planning Library
- ASIA: Adaptive 3D Segmentation using Few Image Annotations
- Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO
- RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulation
- MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
- SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer
- Articulated Object Reconstruction from Rest-State Observation
- It's Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation
- RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design
- Pure Vision Language Action (VLA) Models: A Comprehensive Survey
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training
- VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video
- No Need for Real 3D: Fusing 2D Vision with Pseudo 3D Representations for Robotic Manipulation Learning
- Right-Side-Out: Learning Zero-Shot Sim-to-Real Garment Reversal
- Object Pose Estimation through Dexterous Touch
- Geometric Red-Teaming for Robotic Manipulation
- DYNAMO: Dependency-Aware Deep Learning Framework for Articulated Assembly Motion Prediction
- ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations
- InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layouts
- O3Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation
- ANNIE: Be Careful of Your Robots
- Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots
- U-ARM : Ultra low-cost general teleoperation interface for robot manipulation
- Articulated Object Estimation in the Wild
- Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online
- Integration of Robot and Scene Kinematics for Sequential Mobile Manipulation Planning
- Sensing, Social, and Motion Intelligence in Embodied Navigation: A Comprehensive Survey
- GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects
- Virtual Community: An Open World for Humans, Robots, and Society
- Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
- EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videos
- PCHands: PCA-based Hand Pose Synergy Representation on Manipulators with N-DoF
- CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios
- TacMan-Turbo: Proactive Tactile Control for Robust and Efficient Articulated Object Manipulation
- ScrewSplat: An End-to-End Method for Articulated Object Recognition
- Guiding Diffusion-Based Articulated Object Generation by Partial Point Cloud Alignment and Physical Plausibility Constraints
- DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments
- Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding
- VLMgineer: Vision Language Models as Robotic Toolsmiths
- PhysX-3D: Physical-Grounded 3D Asset Generation
- Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning
- DreamArt: Generating Interactable Articulated Objects from a Single Image
- Integrating Diffusion-based Multi-task Learning with Online Reinforcement Learning for Robust Quadruped Robot Control
- EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modeling
- PRISM: Pointcloud Reintegrated Inference via Segmentation and Cross-attention for Manipulation
Related