Ego-Vision World Model for Humanoid Contact Planning

2025/10/13 by Liu, Hang, Gao, Yuman, Teng, Sangli +5 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2510.11682

Abstract

Enabling humanoid robots to exploit physical contact, rather than simply avoid collisions, is crucial for autonomy in unstructured environments. Traditional optimization-based planners struggle with contact complexity, while on-policy reinforcement learning (RL) is sample-inefficient and has limited multi-task ability. We propose a framework combining a learned world model with sampling-based Model Predictive Control (MPC), trained on a demonstration-free offline dataset to predict future outcomes in a compressed latent space. To address sparse contact rewards and sensor noise, the MPC uses a learned surrogate value function for dense, robust planning. Our single, scalable model supports contact-aware tasks, including wall support after perturbation, blocking incoming objects, and traversing height-limited arches, with improved data efficiency and multi-task capability over on-policy RL. Deployed on a physical humanoid, our system achieves robust, real-time contact planning from proprioception and ego-centric depth images. Website: https://ego-vcp.github.io/

Citations

• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation
• Discrete-Time Hybrid Automata Learning: Legged Locomotion Meets Skateboarding
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics
• AnyCar to Anywhere: Learning Universal Dynamics Model for Agile and Adaptive Mobility
• Full-Order Sampling-Based MPC for Torque-Level Locomotion Control via Diffusion-Style Annealing
• AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation
• WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contacts
• Model-Based Diffusion for Trajectory Optimization
• Genie: Generative Interactive Environments
• Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control
• Extreme Parkour with Legged Robots
• Real-World Humanoid Locomotion with Reinforcement Learning
• Mastering Diverse Domains through World Models
• Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior
• Rapid Locomotion via Reinforcement Learning
• Temporal Difference Learning for Model Predictive Control
• Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
• Learning to Jump from Pixels
• Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement\n Learning
• A Unified MPC Framework for Whole-Body Dynamic Locomotion and Manipulation
• Exploring Simple Siamese Representation Learning
• Exploring Simple Siamese Representation Learning
• Learning Latent Dynamics for Planning from Pixels
• Feedback Control of a Cassie Bipedal Robot: Walking, Standing, and Riding a Segway
• Recurrent World Models Facilitate Policy Evolution
• Proximal Policy Optimization Algorithms

Cited by

• CHyLL: Learning Continuous Neural Representations of Hybrid Systems

Related

• Robust Humanoid Contact Planning with Learned Zero- and One-Step Capturability Prediction
• Embrace Collisions: Humanoid Shadowing for Deployable Contact-Agnostics Motions
• Efficient Humanoid Contact Planning using Learned Centroidal Dynamics Prediction
• Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation
• DROP: Dexterous Reorientation via Online Planning