On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline

2022/12/12 by Nicklas Hansen, Hansen, Nicklas, Zhecheng Yuan +13 · 6 citations
Computer Science · #Advanced Neural Network Applications #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2212.05749

openalex publication_date 2022/12/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we examine the effectiveness of pre-training for visuo-motor control tasks. We revisit a simple Learning-from-Scratch (LfS) baseline that incorporates data augmentation and a shallow ConvNet, and find that this baseline is surprisingly competitive with recent approaches (PVR, MVP, R3M) that leverage frozen visual representations trained on large-scale vision datasets -- across a variety of algorithms, task domains, and metrics in simulation and on a real robot. Our results demonstrate that these methods are hindered by a significant domain gap between the pre-training datasets and current benchmarks for visuo-motor control, which is alleviated by finetuning. Based on our findings, we provide recommendations for future research in pre-training for control and hope that our simple yet strong baseline will aid in accurately benchmarking progress in this area.

Cited by

• IGen: Scalable Data Generation for Robot Learning from Open-World Images
• Learning Massively Multitask World Models for Continuous Control
• SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation
• Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cues
• Exploring Conditions for Diffusion models in Robotic Control
• Maximum In-Support Return Modeling for Dynamic Recommendation with Language Model Prior

Related

• An Unbiased Look at Datasets for Visuo-Motor Pre-Training
• Masked Visual Pre-training for Motor Control
• Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods
• For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equal
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives