Ranking Policy Gradient

2019/06/24 by Lin, Kaixiang, Zhou, Jiayu
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1906.09674

Abstract

Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency. Furthermore, the sample complexity of RPG does not depend on the dimension of state space, which enables RPG for large-scale problems. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.

Related

• Policy Gradient for Reinforcement Learning with General Utilities
• Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling
• Proximal Policy Optimization Algorithms