Adaptive Trade-Offs in Off-Policy Learning

2019/10/16 by Rowland, Mark, Dabney, Will, Munos, Rémi
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1910.07478

Abstract

A great variety of off-policy learning algorithms exist in the literature, and new breakthroughs in this area continue to be made, improving theoretical understanding and yielding state-of-the-art reinforcement learning algorithms. In this paper, we take a unifying view of this space of algorithms, and consider their trade-offs of three fundamental quantities: update variance, fixed-point bias, and contraction rate. This leads to new perspectives of existing methods, and also naturally yields novel algorithms for off-policy evaluation and control. We develop one such algorithm, C-trace, demonstrating that it is able to more efficiently make these trade-offs than existing methods in use, and that it can be scaled to yield state-of-the-art performance in large-scale environments.

Related

• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• Handling Cost and Constraints with Off-Policy Deep Reinforcement Learning
• Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
• Safe and Efficient Off-Policy Reinforcement Learning
• Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift