Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

2020/04/28 by Ilya Kostrikov, Kostrikov, Ilya, Denis Yarats +3 · 31 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2004.13649

openalex publication_date 2020/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function. Existing model-free approaches, such as Soft Actor-Critic (SAC), are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC's performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based (Dreamer, PlaNet, and SLAC) methods and recently proposed contrastive learning (CURL). Our approach can be combined with any model-free reinforcement learning algorithm, requiring only minor modifications. An implementation can be found at https://sites.google.com/view/data-regularized-q.

Cited by

• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• Spectral Representation-based Reinforcement Learning
• AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis
• See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
• Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning
• Learning Massively Multitask World Models for Continuous Control
• Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Reinforcement Learning Using known Invariances
• LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation
• SymmGrid: Super-Scaling On-Robot Learning with Parallelized Symmetries and Egocentric-Exocentric Visual Perception
• MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal Reasoning
• CDE: Concept-Driven Exploration for Reinforcement Learning
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies
• An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Residual Off-Policy RL for Finetuning Behavior Cloning Policies
• Reinforcement Learning for Robotic Insertion of Flexible Cables in Industrial Settings
• Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)
• DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation
• Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator
• LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning
• Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
• Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation
• SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• One-Step Flow Policy Mirror Descent

Related

• Revisiting Data Augmentation in Deep Reinforcement Learning
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Fully Convolutional Network with Multi-Step Reinforcement Learning for Image Processing
• Reinforcement Learning with Augmented Data
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization