Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

2020/12/14 by Albert Zhan, Zhan, Albert, Lerrel Pinto +6 · 3 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2012.07975

openalex publication_date 2020/12/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in unsupervised representation learning significantly improved the sample efficiency of training Reinforcement Learning policies in simulated environments. However, similar gains have not yet been seen for real-robot reinforcement learning. In this work, we focus on enabling data-efficient real-robot learning from pixels. We present Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER), a method that utilizes data augmentation and unsupervised learning to achieve sample-efficient training of real-robot arm policies from sparse rewards. While contrastive pre-training, data augmentation, demonstrations, and reinforcement learning are alone insufficient for efficient learning, our main contribution is showing that the combination of these disparate techniques results in a simple yet data-efficient method. We show that, given only 10 demonstrations, a single robotic arm can learn sparse-reward manipulation policies from pixels, such as reaching, picking, moving, pulling a large object, flipping a switch, and opening a drawer in just 30 minutes of mean real-world training time. We include videos and code on the project website: https://sites.google.com/view/efficient-robotic-manipulation/home

Citations

• Trust Region Policy Optimization
• Gradient-based learning applied to document recognition
• End-to-End Training of Deep Visuomotor Policies
• Continuous control with deep reinforcement learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• DeepMind Control Suite
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Robot Learning in Homes: Improving Generalization and Reducing Dataset\n Bias
• Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data\n for Imitation
• Soft Actor-Critic Algorithms and Applications
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Reinforcement Learning with Augmented Data
• Momentum Contrast for Unsupervised Visual Representation Learning
• Decoupling Representation Learning from Reinforcement Learning
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Cited by

• M³A Policy: Mutable Material Manipulation Augmentation Policy through Photometric Re-rendering
• Reinforcement Learning for Robotic Insertion of Flexible Cables in Industrial Settings
• Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision

Related

• Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods
• Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
• Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation
• DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation