GRS: Generating Robotic Simulation Tasks from Real-World Images

2024/10/20 by Alex Zook, Fan-Yun Sun, Zook, Alex +9 · 4 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Manufacturing Process and Optimization #Robotic Path Planning Algorithms #Robotics (cs.RO) #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.2410.15536

openalex publication_date 2024/10/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training. Using vision-language models (VLMs), our pipeline operates in three stages: 1) scene comprehension with SAM2 for segmentation and object description, 2) matching objects with simulation-ready assets, and 3) generating appropriate tasks. We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code. Experiments demonstrate our system's effectiveness in object correspondence and task environment generation through our novel router mechanism.

Cited by

• TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
• IGen: Scalable Data Generation for Robot Learning from Open-World Images
• Robot Learning from a Physical World Model
• Robot Learning from Any Images

Related

• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
• GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation
• URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images
• REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
• Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation