Convergent and Efficient Deep Q Network Algorithm

2021/06/29 by Zhikang T. Wang, Masahito Ueda, Wang, Zhikang T. +1
Computer Science · #Adaptive Dynamic Programming Control #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2106.15419

openalex publication_date 2021/06/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the empirical success of the deep Q network (DQN) reinforcement learning algorithm and its variants, DQN is still not well understood and it does not guarantee convergence. In this work, we show that DQN can indeed diverge and cease to operate in realistic settings. Although there exist gradient-based convergent methods, we show that they actually have inherent problems in learning dynamics which cause them to fail even in simple tasks. To overcome these problems, we propose a convergent DQN algorithm (C-DQN) that is guaranteed to converge and can work with large discount factors (0.9998). It learns robustly in difficult settings and can learn several difficult games in the Atari 2600 benchmark that DQN fails to solve. Our codes have been publicly released and can be used to reproduce our results.

Citations

• Continuous control with deep reinforcement learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Reinforcement Learning from Imperfect Demonstrations
• Deep Reinforcement Learning and the Deadly Triad
• Towards Characterizing Divergence in Deep Q-Learning
• Gradient Temporal-Difference Learning with Regularized Corrections

Related

• Implementing the Deep Q-Network
• A Theoretical Analysis of Deep Q-Learning
• Deep Reinforcement Learning with Double Q-learning
• M²DQN: A Robust Method for Accelerating Deep Q-learning Network
• Does DQN Learn?