Deep Reinforcement Learning with Robust and Smooth Policy

2020/03/21 by Qianli Shen, Yan Li, Shen, Qianli +7
Computer Science · Engineering · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Fuel Cells and Related Materials #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2003.09534

ICML 2020

openalex publication_date 2020/03/21 · arxiv created 2020/08/15 · arxiv updated 2020/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep reinforcement learning (RL) has achieved great empirical successes in various domains. However, the large search space of neural networks requires a large amount of data, which makes the current RL algorithms not sample efficient. Motivated by the fact that many environments with continuous state space have smooth transitions, we propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- Smooth Regularized Reinforcement Learning (SR²L), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy. Moreover, our proposed framework can also improve the robustness of policy against measurement error in the state space, and can be naturally extended to distribubutionally robust setting. We apply the proposed framework to both on-policy (TRPO) and off-policy algorithm (DDPG). Through extensive experiments, we demonstrate that our method achieves improved sample efficiency and robustness.

Citations

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Continuous control with deep reinforcement learning
• Distributionally Robust Stochastic Optimization with Wasserstein Distance
• Robust Adversarial Reinforcement Learning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Understanding deep learning requires rethinking generalization
• Unsupervised Data Augmentation for Consistency Training
• Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Related

• Regularization Matters in Policy Optimization
• StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent
• Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
• State Regularized Policy Optimization on Data with Dynamics Shift
• The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations