EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations

2025/10/31 by Justin Yu, Yu, Justin, Yide Shentu +9 · 2 citations
Engineering · Neuroscience · Psychology · #Active vision #FOS: Computer and information sciences #Fixation (population genetics) #Head (geology) #Human–robot interaction #Imitation #Motion capture #Motor Control and Adaptation #Robot #Robot Manipulation and Learning #Robot learning #Robotics (cs.RO) #Social Robot Interaction and HRI

paper · pdf · doi:10.48550/arxiv.2511.00153

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/31 · openalex created_date 2025/11/05 · openalex updated_date 2026/08/05

Abstract

Imitation learning from human demonstrations offers a promising approach for robot skill acquisition, but egocentric human data introduces fundamental challenges due to the embodiment gap. During manipulation, humans actively coordinate head and hand movements, continuously reposition their viewpoint and use pre-action visual fixation search strategies to locate relevant objects. These behaviors create dynamic, task-driven head motions that static robot sensing systems cannot replicate, leading to a significant distribution shift that degrades policy performance. We present EgoMI (Egocentric Manipulation Interface), a framework that captures synchronized end-effector and active head trajectories during manipulation tasks, resulting in data that can be retargeted to compatible semi-humanoid robot embodiments. To handle rapid and wide-spanning head viewpoint changes, we introduce a memory-augmented policy that selectively incorporates historical observations. We evaluate our approach on a bimanual robot equipped with an actuated camera head and find that policies with explicit head-motion modeling consistently outperform baseline methods. Results suggest that coordinated hand-eye learning with EgoMI effectively bridges the human-robot embodiment gap for robust imitation learning on semi-humanoid embodiments. Project page: https://egocentric-manipulation-interface.github.io

Citations

• Viser: Imperative, Web-based 3D Visualization in Python
• Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers
• Vision in Action: Learning Active Perception from Human Demonstrations
• Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop
• Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better
• EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
• PyRoki: A Modular Toolkit for Robot Kinematic Optimization
• π_0.5: a Vision-Language-Action Model with Open-World Generalization
• AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletons
• Bimanual Dexterity for Complex Tasks
• EgoMimic: Scaling Imitation Learning via Egocentric Video
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback
• OpenVLA: An Open-Source Vision-Language-Action Model
• Octo: An Open-Source Generalist Robot Policy
• DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
• Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
• Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation
• On Bringing Robots Home
• Vision-Language Foundation Models as Effective Robot Imitators
• GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
• Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• RT-1: Robotics Transformer for Real-World Control at Scale
• Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
• VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors
• Interactive Language: Talking to Robots in Real Time
• Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
• Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube
• Visual Imitation Made Easy
• Language Conditioned Imitation Learning over Unstructured Data
• Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
• DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
• On the Continuity of Rotation Representations in Neural Networks

Cited by

• OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction
• Thinking in 360°: Humanoid Visual Search in the Wild

Related

• EgoMimic: Scaling Imitation Learning via Egocentric Video
• EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos
• EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World
• EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Data
• EgoDemoGen: Novel Egocentric Demonstration Generation Enables Viewpoint-Robust Manipulation