Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

2020/04/28 by Ilya Kostrikov, Kostrikov, Ilya, Denis Yarats +3 · 70 citations
Computer Science · Engineering · Mathematics · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.CV #cs.LG #eess.IV #electronic engineering #information engineering #stat.ML

paper · pdf · doi:10.48550/arxiv.2004.13649

openalex publication_date 2020/04/28 · arxiv created 2021/03/07 · arxiv updated 2021/03/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to regularize the value function. Existing model-free approaches, such as Soft Actor-Critic (SAC), are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC's performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based (Dreamer, PlaNet, and SLAC) methods and recently proposed contrastive learning (CURL). Our approach can be combined with any model-free reinforcement learning algorithm, requiring only minor modifications. An implementation can be found at https://sites.google.com/view/data-regularized-q.

Cited by

• Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
• Spectral Representation-based Reinforcement Learning
• AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis
• See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
• Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning
• Learning Massively Multitask World Models for Continuous Control
• Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Reinforcement Learning Using known Invariances
• LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation
• SymmGrid: Super-Scaling On-Robot Learning with Parallelized Symmetries and Egocentric-Exocentric Visual Perception
• MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning
• CDE: Concept-Driven Exploration for Reinforcement Learning
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies
• An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Residual Off-Policy RL for Finetuning Behavior Cloning Policies
• Reinforcement Learning for Robotic Insertion of Flexible Cables in Industrial Settings
• Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)
• DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation
• Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator
• LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning
• Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
• Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation
• SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• One-Step Flow Policy Mirror Descent
• MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
• MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
• Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning
• EXPO: Stable Reinforcement Learning with Expressive Policies
• rQdia: Regularizing Q-Value Distributions With Image Augmentation
• Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces
• Dream to Generalize: Zero-Shot Model-Based Reinforcement Learning for Unseen Visual Distractions
• Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning
• Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulation
• Simple, Good, Fast: Self-Supervised World Models Free of Baggage
• Optimistic critics can empower small actors
• State-Covering Trajectory Stitching for Diffusion Planners
• RoboTransfer: Geometry-Consistent Video Diffusion for Robotic Visual Policy Transfer
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
• Grounding Bodily Awareness in Visual Representations for Efficient Policy Learning
• Enter the Void - Planning to Seek Entropy When Reward is Scarce
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
• Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning
• Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
• Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors
• Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
• Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
• Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
• D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentation
• Reward-Conditioned Reinforcement Learning
• Next Embedding Prediction Makes World Models Stronger
• Maximum Likelihood Reinforcement Learning
• χ₀: Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies
• EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning
• Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images
• Accelerating Visual Reinforcement Learning with Separate Primitive Policy for Peg-in-Hole Tasks
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
• Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation
• Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
• MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learning
• Adaptive Insurance Reserving with CVaR-Constrained Reinforcement Learning under Macroeconomic Regimes
• Deep Reinforcement Learning via Object-Centric Attention

Related

• Revisiting Data Augmentation in Deep Reinforcement Learning
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Fully Convolutional Network with Multi-Step Reinforcement Learning for Image Processing
• Reinforcement Learning with Augmented Data
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization