Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning

2021/05/14 by Fan Zhou, Zhou, Fan, Zhoufan Zhu +5
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2105.06696

openalex created_date 2021/02/15 · openalex publication_date 2021/05/14 · openalex updated_date 2026/07/28

Abstract

Although distributional reinforcement learning (DRL) has been widely examined in the past few years, there are two open questions people are still trying to address. One is how to ensure the validity of the learned quantile function, the other is how to efficiently utilize the distribution information. This paper attempts to provide some new perspectives to encourage the future in-depth studies in these two fields. We first propose a non-decreasing quantile function network (NDQFN) to guarantee the monotonicity of the obtained quantile estimates and then design a general exploration framework called distributional prediction error (DPE) for DRL which utilizes the entire distribution of the quantile function. In this paper, we not only discuss the theoretical necessity of our method but also show the performance gain it achieves in practice by comparing with some competitors on Atari 2600 Games especially in some hard-explored games.

Citations

• Playing Atari with Deep Reinforcement Learning
• Robust Estimation of a Location Parameter
• Prioritized Experience Replay
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Rainbow: Combining Improvements in Deep Reinforcement Learning

Related

• Fully Parameterized Quantile Function for Distributional Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Cramér Distance perspective on Quantile Regression based Distributional Reinforcement Learning
• Deep Distributional Learning with Non-crossing Quantile Network