Adapting Double Q-Learning for Continuous Reinforcement Learning

2023/09/25 by Kuznetsov, Arsenii
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2309.14471

Abstract

Majority of off-policy reinforcement learning algorithms use overestimation bias control techniques. Most of these techniques rooted in heuristics, primarily addressing the consequences of overestimation rather than its fundamental origins. In this work we present a novel approach to the bias correction, similar in spirit to Double Q-Learning. We propose using a policy in form of a mixture with two components. Each policy component is maximized and assessed by separate networks, which removes any basis for the overestimation bias. Our approach shows promising near-SOTA results on a small set of MuJoCo environments.

Related

• Smoothed Q-learning
• Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
• ADDQ: Adaptive Distributional Double Q-Learning
• Cross Learning in Deep Q-Networks
• Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning