2024/10/25 by Zhou, Liyou, Oleg Sinavski, Sinavski, Oleg +2
Computer Science · Engineering · #Artificial intelligence #Computer science #Human–computer interaction #Open source #Operating system #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Simulation #Software
paper · pdf · doi:10.48550/arxiv.2410.19564
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The emergence of 3D Gaussian Splatting for fast and high-quality novel view synthesize has opened up the possibility to construct photo-realistic simulations from video for robotic reinforcement learning. While the approach has been demonstrated in several research papers, the software tools used to build such a simulator remain unavailable or proprietary. We present SplatGym, an open source neural simulator for training data-driven robotic control policies. The simulator creates a photorealistic virtual environment from a single video. It supports ego camera view generation, collision detection, and virtual object in-painting. We demonstrate training several visual navigation policies via reinforcement learning. SplatGym represents a notable first step towards an open-source general-purpose neural environment for robotic learning. It broadens the range of applications that can effectively utilise reinforcement learning by providing convenient and unrestricted tooling, and by eliminating the need for the manual development of conventional 3D environments.