Differentially Private Exploration in Reinforcement Learning with Linear Representation

2021/12/02 by Luyo, Paul, Garcelon, Evrard, Lazaric, Alessandro +1
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2112.01585

Abstract

This paper studies privacy-preserving exploration in Markov Decision Processes (MDPs) with linear representation. We first consider the setting of linear-mixture MDPs (Ayoub et al., 2020) (a.k.a. model-based setting) and provide an unified framework for analyzing joint and local differential private (DP) exploration. Through this framework, we prove a \widetildeO(K^3/4/√ε) regret bound for (ε,δ)-local DP exploration and a \widetildeO(√(K/ε)) regret bound for (ε,δ)-joint DP. We further study privacy-preserving exploration in linear MDPs (Jin et al., 2020) (a.k.a. model-free setting) where we provide a \widetildeO(K^(3)/(5)/ε^(2)/(5)) regret bound for (ε,δ)-joint DP, with a novel algorithm based on low-switching. Finally, we provide insights into the issues of designing local DP algorithms in this model-free setting.

Related

• Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processes
• Towards Optimal Differentially Private Regret Bounds in Linear MDPs
• Differentially Private Reinforcement Learning with Linear Function Approximation
• Improved Regret for Differentially Private Exploration in Linear MDP
• Near-Optimal Differentially Private Reinforcement Learning