Relative Entropy Regularized Reinforcement Learning for Efficient Encrypted Policy Synthesis

2025/06/14 by Suh, Jihoon, Jang, Yeongjun, Teranishi, Kaoru +1
#FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2506.12358

Abstract

We propose an efficient encrypted policy synthesis to develop privacy-preserving model-based reinforcement learning. We first demonstrate that the relative-entropy-regularized reinforcement learning framework offers a computationally convenient linear and ``min-free'' structure for value iteration, enabling a direct and efficient integration of fully homomorphic encryption with bootstrapping into policy synthesis. Convergence and error bounds are analyzed as encrypted policy synthesis propagates errors under the presence of encryption-induced errors including quantization and bootstrapping. Theoretical analysis is validated by numerical simulations. Results demonstrate the effectiveness of the RERL framework in integrating FHE for encrypted policy synthesis.

Citations

• Ring-LWE based encrypted controller with unlimited number of recursive multiplications and effect of error growth
• Encrypted extremum seeking for privacy-preserving PID tuning as-a-Service
• Encrypted dynamic control with unlimited operating time via FIR filters
• Encrypted Value Iteration and Temporal Difference Learning over Leveled Homomorphic Encryption
• SARSA(0) Reinforcement Learning over Fully Homomorphic Encryption
• Dynamic controller that operates over homomorphically encrypted data for infinite time horizon
• Cloud-based Quadratic Optimization with Partially Homomorphic Encryption
• Cloud-based MPC with Encrypted Data

Related

• Encrypted Value Iteration and Temporal Difference Learning over Leveled Homomorphic Encryption
• Reinforcement Learning on Encrypted Data
• Regularized Policies are Reward Robust
• An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning
• EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization