Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces

2020/02/07 by Bogdan Mazoure, Thang Doan, Mazoure, Bogdan +12
Computer Science · Mathematics · Physics and Astronomy · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Reinforcement Learning in Robotics #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2002.02863

openalex publication_date 2020/02/07 · arxiv created 2020/10/15 · arxiv updated 2020/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a general framework for policy representation for reinforcement learning tasks. This framework involves finding a low-dimensional embedding of the policy on a reproducing kernel Hilbert space (RKHS). The usage of RKHS based methods allows us to derive strong theoretical guarantees on the expected return of the reconstructed policy. Such guarantees are typically lacking in black-box models, but are very desirable in tasks requiring stability. We conduct several experiments on classic RL domains. The results confirm that the policies can be robustly embedded in a low-dimensional space while the embedded policy incurs almost no decrease in return.

Citations

• Comparison of perturbation bounds for the stationary distribution of a Markov chain
• Orthonormal bases of compactly supported wavelets
• Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
• Hilbert Space Embeddings of POMDPs
• Continuous control with deep reinforcement learning
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Robust Adversarial Reinforcement Learning
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Related

• Policy Newton Algorithm in Reproducing Kernel Hilbert Space
• Vector-Valued Distributional Reinforcement Learning Policy Evaluation: A Hilbert Space Embedding Approach
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spaces
• Representation-Driven Reinforcement Learning