Control Synthesis with Reinforcement Learning: A Modeling Perspective

2025/10/29 by Xu, Nikki, Tran, Hien
#FOS: Electrical engineering #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2510.25063

Abstract

Controllers designed with reinforcement learning can be sensitive to model mismatch. We demonstrate that designing such controllers in a virtual simulation environment with an inaccurate model is not suitable for deployment in a physical setup. Controllers designed using an accurate model is robust against disturbance and small mismatch between the physical setup and the mathematical model derived from first principles; while a poor model results in a controller that performs well in simulation but fails in physical experiments. Sensitivity analysis is used to justify these discrepancies and an empirical region of attraction estimation help us visualize their robustness.

Citations

• Symmetry in Neural Network Parameter Spaces
• Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies
• AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images
• Examining the simulation-to-reality gap of a wheel loader digging in deformable terrain
• Tight Certified Robustness via Min-Max Representations of ReLU Neural Networks
• Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization
• Residual Physics Learning and System Identification for Sim-to-real Transfer of Policies on Buoyancy Assisted Legged Robots
• Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
• Robot Learning from Randomized Simulations: A Review
• Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning
• Auto-Tuned Sim-to-Real Transfer
• Chance Constrained Policy Optimization for Process Control and Optimization
• Control Barrier Functions: Theory and Applications
• The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems
• A Tour of Reinforcement Learning: The View from Continuous Control
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
• Variance-based sensitivity analysis for time-dependent processes
• Robust Adversarial Reinforcement Learning
• Towards Generalization and Simplicity in Continuous Control
• Continuous control with deep reinforcement learning
• Distributionally Robust Markov Decision Processes
• Robust Dynamic Programming
• Reinforcement Learning: An Introduction

Related

• Unlocking the Potential of Simulators: Design with RL in Mind
• Reinforcement learning for non-prehensile manipulation: Transfer from\n simulation to physical system
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Counterexample-Guided Synthesis of Perception Models and Control
• Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics