Domain Randomization for Transferring Deep Neural Networks from\n Simulation to the Real World

2017/03/20 by Josh Tobin, Rachel Fong, Tobin, Josh +11 · 190 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.1703.06907

Abstract

Bridging the 'reality gap' that separates simulated robotics from experiments\non hardware could accelerate robotic research through improved data\navailability. This paper explores domain randomization, a simple technique for\ntraining models on simulated images that transfer to real images by randomizing\nrendering in the simulator. With enough variability in the simulator, the real\nworld may appear to the model as just another variation. We focus on the task\nof object localization, which is a stepping stone to general robotic\nmanipulation skills. We find that it is possible to train a real-world object\ndetector that is accurate to 1.5cm and robust to distractors and partial\nocclusions using only data from a simulator with non-realistic random textures.\nTo demonstrate the capabilities of our detectors, we show they can be used to\nperform grasping in a cluttered environment. To our knowledge, this is the\nfirst successful transfer of a deep neural network trained only on simulated\nRGB images (without pre-training on real images) to the real world for the\npurpose of robotic control.\n

Citations

• Deep Domain Confusion: Maximizing for Domain Invariance
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

Cited by

• VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
• Learning Adaptive Multi-Task Guidance, Navigation, and Control via Hypernetworks
• The Curse of Precision: A Data Scaling Law for High-Precision Robotic Manipulation
• Meshless Domain Randomization via Explicit Parameter Perturbation of 3D Gaussian Splatting
• Learning the Macroeconomic Language
• Learning Generalizable Hand-Object Tracking from Synthetic Demonstrations
• Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments
• When Robots Say No: The Empathic Ethical Disobedience Benchmark
• Bridging simulation and reality in subsurface radar-based sensing: physics-guided hierarchical domain adaptation with deep adversarial learning
• Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery
• NDRL: Cotton Irrigation and Nitrogen Application with Nested Dual-Agent Reinforcement Learning
• Synthetic Data Pipelines for Adaptive, Mission-Ready Militarized Humanoids
• AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis
• Expert Switching for Robust AAV Landing: A Dual-Detector Framework in Simulation
• Computer vision training dataset generation for robotic environments using Gaussian splatting
• Fast Policy Learning for 6-DOF Position Control of Underwater Vehicles
• State-Conditional Adversarial Learning: An Off-Policy Visual Domain Transfer Method for End-to-End Imitation Learning
• Task-Oriented Grasping Using Reinforcement Learning with a Contextual Reward Machine
• Invariance Co-training for Robot Visual Generalization
• Model-Free Assessment of Simulator Fidelity via Quantile Curves
• Task diversity produces systematic transfer but inhibits continual reinforcement learning
• Bridging Simulation and Reality: Cross-Domain Transfer with Semantic 2D Gaussian Splatting
• Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia
• Vehicle Dynamics Embedded World Models for Autonomous Driving
• Data-Centric Visual Development for Self-Driving Labs
• Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
• DiG-Flow: Discrepancy-Guided Flow Matching for Robust VLA Models
• FOD-S2R: A FOD Dataset for Sim2Real Transfer Learning based Object Detection
• M³A Policy: Mutable Material Manipulation Augmentation Policy through Photometric Re-rendering
• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
• H-Zero: Cross-Humanoid Locomotion Pretraining Enables Few-shot Novel Embodiment Transfer
• MILE: A Mechanically Isomorphic Exoskeleton Data Collection System with Fingertip Visuotactile Sensing for Dexterous Manipulation
• Synthetic Industrial Object Detection: GenAI vs. Feature-Based Methods
• Hybrid Synthetic Data Generation with Domain Randomization Enables Zero-Shot Vision-Based Part Inspection Under Extreme Class Imbalance
• Improving Robotic Manipulation Robustness via NICE Scene Surgery
• DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization
• Physics-Informed Spiking Neural Networks via Conservative Flux Quantization
• OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
• Arcadia: Toward a Full-Lifecycle Framework for Embodied Lifelong Learning
• AI/ML based Joint Source and Channel Coding for HARQ-ACK Payload
• Domain Randomization for Scene-Specific Car Detection and Pose\n Estimation
• CNN-Based Camera Pose Estimation and Localisation of Scan Images for Aircraft Visual Inspection
• Whole-Body Inverse Dynamics MPC for Legged Loco-Manipulation
• AutoFocus-IL: VLM-based Saliency Maps for Data-Efficient Visual Imitation Learning without Extra Human Annotations
• Quantifying Generalization in Reinforcement Learning
• DynaMimicGen: A Data Generation Framework for Robot Learning of Dynamic Tasks
• Scalable agent alignment via reward modeling: a research direction
• Bi-AQUA: Bilateral Control-Based Imitation Learning for Underwater Robot Arms via Lighting-Aware Action Chunking with Transformers
• DiffuDepGrasp: Diffusion-based Depth Noise Modeling Empowers Sim2Real Robotic Grasping
• Physics-Based Benchmarking Metrics for Multimodal Synthetic Images
• VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation
• X-WIN: Building Chest Radiograph World Model via Predictive Sensing
• An Operational Kardashev-Style Scale for Autonomous AI - Towards AGI and Superintelligence
• Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs
• Auto-Tuned Sim-to-Real Transfer
• Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation
• Scalable Policy Evaluation with Video World Models
• Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective
• SPIDER: Scalable Physics-Informed Dexterous Retargeting
• Data Fusion-Enhanced Decision Transformer for Stable Cross-Domain Generalization
• Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy
• RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Models
• Adaptive PID Control for Robotic Systems via Hierarchical Meta-Learning and Reinforcement Learning with Physics-Based Data Augmentation
• Perturbation-mitigated USV Navigation with Distributionally Robust Reinforcement Learning
• Distributionally Robust Self Paced Curriculum Reinforcement Learning
• SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
• Scaling Multi-Agent Environment Co-Design with Diffusion Models
• Collaborative Assembly Policy Learning of a Sightless Robot
• Heuristic Adaptation of Potentially Misspecified Domain Support for Likelihood-Free Inference in Stochastic Dynamical Systems
• Adaptive Inverse Kinematics Framework for Learning Variable-Length Tool Manipulation in Robotics
• Reinforcement Learning for Robotic Safe Control with Force Sensing
• Morphology-Aware Graph Reinforcement Learning for Tensegrity Robot Locomotion
• Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning
• Avoidance of Manual Labeling in Robotic Autonomous Navigation Through Multi-Sensory Semi-Supervised Learning
• Reverse Curriculum Generation for Reinforcement Learning
• ORRB -- OpenAI Remote Rendering Backend
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
• Applying Domain Randomization to Synthetic Data for Object Category Detection
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Hindsight Experience Replay
• Advancing vegetation monitoring with virtual laser scanning of dynamic scenes ( <scp>VLS</scp> ‐ <scp>4D</scp> ): Opportunities, implementations and future perspectives
• TARC: Time-Adaptive Robotic Control
• Reliable Robotic Task Execution in the Face of Anomalies
• EasyUUV: An LLM-Enhanced Universal and Lightweight Sim-to-Real Reinforcement Learning Framework for UUV Attitude Control
• Combining SAR Simulators to Train ATR Models with Synthetic Data
• Domain Generalization with MixStyle
• Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization
• Synthetic Data for Robust Runway Detection
• The Reality Gap in Robotics: Challenges, Solutions, and Best Practices
• Efficient Model-Based Reinforcement Learning for Robot Control via Online Optimization
• Closing the Sim2Real Performance Gap in RL
• RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation
• Adversarial Reinforcement Learning for Robust Control of Fixed-Wing Aircraft under Model Uncertainty
• RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
• Procedural Game Level Design with Deep Reinforcement Learning
• Leveraging Neural Descriptor Fields for Learning Contact-Aware Dynamic Recovery
• Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotion
• ViTacGen: Robotic Pushing with Vision-to-Touch Generation
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Bridge the Gap: Enhancing Quadruped Locomotion with Vertical Ground Perturbations
• Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control
• Learning to Grasp Anything by Playing with Random Toys
• Learning Robust Agile Flight Control with Stability Guarantees
• Phys2Real: Fusing VLM Priors with Interactive Online Adaptation for Uncertainty-Aware Sim-to-Real Manipulation
• SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy
• PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System
• High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting
• PolySim: Bridging the Sim-to-Real Gap for Humanoid Control via Multi-Simulator Dynamics Randomization
• Constrained Natural Language Action Planning for Resilient Embodied Systems
• Discovering Blind Spots in Reinforcement Learning
• Model-Based Adaptive Precision Control for Tabletop Planar Pushing Under Uncertain Dynamics
• CVSM: Contrastive Vocal Similarity Modeling
• Generation of GelSight Tactile Images for Sim2Real Learning
• Best of Sim and Real: Decoupled Visuomotor Manipulation via Learning Control in Simulation and Perception in Real
• Data-Efficient Multitask DAgger
• Fidelity-Aware Data Composition for Robust Robot Generalization
• SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics
• Adversarial Diffusion for Robust Reinforcement Learning
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Space Robotics Bench: Robot Learning Beyond Earth
• Multi-Agent Path Finding via Offline RL and LLM Collaboration
• Learning Multi-Skill Legged Locomotion Using Conditional Adversarial Motion Priors
• GraphUniverse: Synthetic Graph Generation for Evaluating Inductive Generalization
• Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
• It's Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation
• Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI
• On the Importance of Multistability for Horizon Generalization in Reinforcement Learning
• Selecting Data Augmentation for Simulating Interventions
• Evaluating Synthetic Data Generation for Domain Generalization in Fetal Brain MRI Segmentation
• SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
• Do You Need Proprioceptive States in Visuomotor Policies?
• Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training
• SINGER: An Onboard Generalist Vision-Language Navigation Policy for Drones
• Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach
• RoboSeek: You Need to Interact with Your Objects
• Recovering Parametric Scenes from Very Few Time-of-Flight Pixels
• LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrations
• Synthetic-to-Real Object Detection using YOLOv11 and Domain Randomization Strategies
• A Synthetic Dataset for Manometry Recognition in Robotic Applications
• VIRTUS-FPP: Virtual Sensor Modeling for Fringe Projection Profilometry in NVIDIA Isaac Sim
• TreeIRL: Safe Urban Driving with Tree Search and Inverse Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Transferring End-to-End Visuomotor Control from Simulation to Real World\n for a Multi-Stage Task
• Time to Play: Simulating Early-Life Animal Dynamics Enhances Robotics Locomotion Discovery
• Scaling Up Forest Vision with Synthetic Data
• Real-time reinforcement learning for turbulent state-dependent control in a bluff-body wake
• RoVerFly: Robust and Versatile Implicit Hybrid Control of Quadrotor-Payload Systems
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration
• Towards scalable organ level 3D plant segmentation: Bridging the data algorithm computing gap
• Simulation Priors for Data-Efficient Deep Learning
• Learning Tool-Aware Adaptive Compliant Control for Autonomous Regolith Excavation
• Learning image-based Receding Horizon Planning for manipulation in clutter
• Online time series prediction using feature adjustment
• CTBC: Contact-Triggered Blind Climbing for Wheeled Bipedal Robots with Instruction Learning and Reinforcement Learning
• DUViN: Diffusion-Based Underwater Visual Navigation via Knowledge-Transferred Depth Features
• Multi-Embodiment Locomotion at Scale with extreme Embodiment Randomization
• Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots
• Fail2Progress: Learning from Real-World Robot Failures with Stein Variational Inference
• Poke and Strike: Learning Task-Informed Exploration Policies
• Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation
• cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending
• Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization
• Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
• Adversarial Discriminative Sim-to-real Transfer of Visuo-motor Policies
• NViSII: A Scriptable Tool for Photorealistic Image Generation
• Universal Planning Networks
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
• Human Centric General Physical Intelligence for Agile Manufacturing Automation
• Sim2Dust: Mastering Dynamic Waypoint Tracking on Granular Media
• Multi-Group Equivariant Augmentation for Reinforcement Learning in Robot Manipulation
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• GBC: Generalized Behavior-Cloning Framework for Whole-Body Humanoid Imitation
• Generative Modeling for Robust Deep Reinforcement Learning on the Traveling Salesman Problem
• Unsupervised Skill Discovery as Exploration for Learning Agile Locomotion
• Learning to Grasp from a Single Demonstration
• Learning a Vision-Based Footstep Planner for Hierarchical Walking Control
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Unsupervised Partner Design Enables Robust Ad-hoc Teamwork
• Static and Plugged: Make Embodied Evaluation Simple
• Physics-Guided Memory Network for Building Energy Modeling
• Learning Pivoting Manipulation with Force and Vision Feedback Using Optimization-based Demonstrations
• Reducing the gap between general purpose data and aerial images in concentrated solar power plants
• Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion
• Improving Generalization Ability of Robotic Imitation Learning by Resolving Causal Confusion in Observations
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
• DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learning
• Autonomous Industrial Management via Reinforcement Learning: Self-Learning Agents for Decision-Making -- A Review
• Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalization

Related

• Robust Visual Sim-to-Real Transfer for Robotic Manipulation
• Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks
• Robot Learning from Randomized Simulations: A Review
• Domain Randomization for Active Pose Estimation
• Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization