Learning Visual Parkour from Generated Images

2024/10/31 by Alan Yu, Alan C. L. Yu, Yu, Alan +10 · 1 voice · 7 citations
Health Professions · Psychology · #Adventure Sports and Sensation Seeking #Physical Education and Training Studies #Recreation, Leisure, Wilderness Management #cs.RO

paper · pdf · doi:10.48550/arxiv.2411.00083

openalex publication_date 2024/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot's ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera. website visit https://lucidsim.github.io

Cited by

• Iterative Compositional Data Generation for Robot Control
• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
• Learning Vision-Driven Reactive Soccer Skills for Humanoid Robots
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
• Residual Off-Policy RL for Finetuning Behavior Cloning Policies
• CObL: Toward Zero-Shot Ordinal Layering without User Prompting
• DISCOVERSE: Efficient Robot Simulation in Complex High-Fidelity Environments

Discussions

• Sharing some new work! A big dream in AI is to create world models of sufficient quality that you can train agents within them. Classic simulators lack visual diversity and realism. GenAI lacks phys [bsky, 48 points, 1 comments]

Related

• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
• Robot Parkour Learning
• Extreme Parkour with Legged Robots
• SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience