Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processes

2021/10/19 by Liao, Chonghua, He, Jiafan, Gu, Quanquan
#Cryptography and Security (cs.CR) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2110.10133

Abstract

Reinforcement learning (RL) algorithms can be used to provide personalized services, which rely on users' private and sensitive data. To protect the users' privacy, privacy-preserving RL algorithms are in demand. In this paper, we study RL with linear function approximation and local differential privacy (LDP) guarantees. We propose a novel (ε, δ)-LDP algorithm for learning a class of Markov decision processes (MDPs) dubbed linear mixture MDPs, and obtains an O( d^5/4H^7/4T^3/4(log(1/δ))^1/4√(1/ε)) regret, where d is the dimension of feature mapping, H is the length of the planning horizon, and T is the number of interactions with the environment. We also prove a lower bound Ω(dH√(T)/(e^ε(e^ε-1))) for learning linear mixture MDPs under ε-LDP constraint. Experiments on synthetic datasets verify the effectiveness of our algorithm. To the best of our knowledge, this is the first provable privacy-preserving RL algorithm with linear function approximation.

Related

• Differentially Private Exploration in Reinforcement Learning with Linear Representation
• Differentially Private Reinforcement Learning with Linear Function Approximation
• Local Differential Privacy for Regret Minimization in Reinforcement Learning
• Towards Optimal Differentially Private Regret Bounds in Linear MDPs
• Near-Optimal Differentially Private Reinforcement Learning