The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation

2023/05/28 by Rowland, Mark, Tang, Yunhao, Lyle, Clare +3 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2305.18388

Abstract

We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.

Cited by

• Auditing the Risk Claims of Distributional Reinforcement Learning

Related

• An Analysis of Quantile Temporal-Difference Learning
• Distributional Reinforcement Learning with Quantile Regression
• Quantile Reinforcement Learning
• Investigating practical linear temporal difference learning
• Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates