Real-World Robot Learning with Masked Visual Pre-training

2022/10/06 by Ilija Radosavovic, Tete Xiao, Radosavovic, Ilija +9 · 49 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Robotics (cs.RO) #cs.CV #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.2210.03109

CoRL 2022; Project page: https://tetexiao.com/projects/real-mvp

arxiv created 2022/10/06 · openalex publication_date 2022/10/06 · arxiv updated 2022/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%). Finally, we train a 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.

Cited by

• Embodied Robot Manipulation in the Era of Foundation Models: Planning and Learning Perspectives
• OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning
• Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
• ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulation
• Improved adaptive wind driven optimization algorithm for real-time path planning
• SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation
• In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data
• Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
• SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulation
• XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations
• Exploring Conditions for Diffusion models in Robotic Control
• Avi: Action from Volumetric Inference
• VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing
• Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress
• Generalist Robot Manipulation beyond Action Labeled Data
• SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
• OpenGVL -- Benchmarking Visual Temporal Progress for Data Curation
• No Need for Real 3D: Fusing 2D Vision with Pseudo 3D Representations for Robotic Manipulation Learning
• 4D Visual Pre-training for Robot Learning
• Self-supervised Learning Of Visual Pose Estimation Without Pose Labels By Classifying LED States
• Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning
• Masquerade: Learning from In-the-wild Human Videos using Data-Editing
• AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
• Video Generators are Robot Policies
• Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
• Latent Policy Steering with Embodiment-Agnostic Pretrained World Models
• UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
• Foundation Model Driven Robotics: A Comprehensive Review
• Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning
• TriVLA: A Triple-System-Based Unified Vision-Language-Action Model with Episodic World Modeling for General Robot Control
• Is an object-centric representation beneficial for robotic manipulation ?
• ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation
• Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning
• Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoning
• Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
• Grounding Bodily Awareness in Visual Representations for Efficient Policy Learning
• Object-Centric Representations Improve Policy Generalization in Robot Manipulation
• EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
• DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies
• Learning Streaming Video Representation via Multitask Training
• CIVIL: Causal and Intuitive Visual Imitation Learning
• SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation
• Dexterous Manipulation through Imitation Learning: A Survey
• Imitation Learning with Precisely Labeled Human Demonstrations
• Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learning
• ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface

Related

• Masked Visual Pre-training for Motor Control
• Robot Learning with Sensorimotor Pre-training
• Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods
• MaskViT: Masked Visual Pre-Training for Video Prediction
• An Unbiased Look at Datasets for Visuo-Motor Pre-Training