2025/02/12 by Xiaoshen Han, Han, Xiaoshen, Minghuan Liu +14 · 24 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Graphics and Visualization Techniques #FOS: Computer and information sciences #Graphics pipeline #Human Motion and Animation #Rendering (computer graphics) #Robot #Robotics #Robotics (cs.RO) #Scalability #Task (project management) #Visualization
paper · pdf · doi:10.48550/arxiv.2502.08645
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/02/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE3SIM, addressing geometric and visual sim-to-real gaps. RE3SIM employs advanced 3D reconstruction and neural rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real pipeline across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58%. To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects. Codes and demos are available at: https://re3sim.github.io/.