Closing the Sim2Real Performance Gap in RL

2025/10/20 by Akhil S Anand, Shambhuraj Sawant, Anand, Akhil S +7 · 1 citation
Computer Science · Decision Sciences · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2510.17709

openalex publication_date 2025/10/20 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28

Abstract

Sim2Real aims at training policies in high-fidelity simulation environments and effectively transferring them to the real world. Despite the developments of accurate simulators and Sim2Real RL approaches, the policies trained purely in simulation often suffer significant performance drops when deployed in real environments. This drop is referred to as the Sim2Real performance gap. Current Sim2Real RL methods optimize the simulator accuracy and variability as proxies for real-world performance. However, these metrics do not necessarily correlate with the real-world performance of the policy as established theoretically and empirically in the literature. We propose a novel framework to address this issue by directly adapting the simulator parameters based on real-world performance. We frame this problem as a bi-level RL framework: the inner-level RL trains a policy purely in simulation, and the outer-level RL adapts the simulation model and in-sim reward parameters to maximize real-world performance of the in-sim policy. We derive and validate in simple examples the mathematical tools needed to develop bi-level RL algorithms that close the Sim2Real performance gap.

Citations

• A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models
• Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification
• Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning
• All AI Models are Wrong, but Some are Optimal
• Optimality Conditions for Model Predictive Control: Rethinking Predictive Model Design
• Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis
• Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning
• A Review of Differentiable Simulators
• Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation
• DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning
• A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning
• Learning Vision-Based Bipedal Locomotion for Challenging Terrain
• Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning
• Value Gradient weighted Model-Based Reinforcement Learning
• Temporal Difference Learning for Model Predictive Control
• Mismatched No More: Joint Model-Policy Optimization for Model-Based RL
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
• The Value Equivalence Principle for Model-Based Reinforcement Learning
• Stochastic Grounded Action Transformation for Robot Learning in\n Simulation
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real
• Language Models are Few-Shot Learners
• MOReL : Model-Based Offline Reinforcement Learning
• Policy-Aware Model Learning for Policy Gradient Methods
• Adversarial-Learned Loss for Domain Adaptation
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Solving Rubik's Cube with a Robot Hand
• The Differentiable Cross-Entropy Method
• When to Trust Your Model: Model-Based Policy Optimization
• Differentiable MPC for End-to-end Planning and Control
• Multi-Adversarial Domain Adaptation
• Learning the Reward Function for a Misspecified Model
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Domain Randomization for Transferring Deep Neural Networks from\n Simulation to the Real World
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Reinforcement Learning: An Introduction

Cited by

• CORL: Reinforcement Learning of MILP Policies Solved via Branch and Bound

Related

• Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
• Sim2real for Reinforcement Learning Driven Next Generation Networks
• Auto-Tuned Sim-to-Real Transfer
• Towards Mitigating Sim2Real Gaps: A Formal Quantitative Approach
• Sim-Anchored Learning for On-the-Fly Adaptation