Robustifying Reinforcement Learning Policies with L₁ Adaptive Control

2021/06/04 by Yikun Cheng, Pan Zhao, Cheng, Yikun +9
Computer Science · Physics and Astronomy · #Adaptive Dynamic Programming Control #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2106.02249

openalex publication_date 2021/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A reinforcement learning (RL) policy trained in a nominal environment could fail in a new/perturbed environment due to the existence of dynamic variations. Existing robust methods try to obtain a fixed policy for all envisioned dynamic variation scenarios through robust or adversarial training. These methods could lead to conservative performance due to emphasis on the worst case, and often involve tedious modifications to the training environment. We propose an approach to robustifying a pre-trained non-robust RL policy with L₁ adaptive control. Leveraging the capability of an L₁ control law in the fast estimation of and active compensation for dynamic variations, our approach can significantly improve the robustness of an RL policy trained in a standard (i.e., non-robust) way, either in a simulator or in the real world. Numerical experiments are provided to validate the efficacy of the proposed approach.

Citations

• Robust Adversarial Reinforcement Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning agile and dynamic motor skills for legged robots
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• L1-Adaptive MPPI Architecture for Robust and Agile Control of Multirotors
• Autonomous Drone Racing with Deep Reinforcement Learning

Related

• Robust Model Based Reinforcement Learning Using L₁ Adaptive Control
• Recovering Robustness in Model-Free Reinforcement learning
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Robust Reinforcement Learning: A Case Study in Linear Quadratic Regulation
• A Crash Course on Reinforcement Learning