Full Gradient Deep Reinforcement Learning for Average-Reward Criterion

2023/04/07 by Pagare, Tejas, Borkar, Vivek, Avrachenkov, Konstantin · 1 citation
#93-06 #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2304.03729

Abstract

We extend the provably convergent Full Gradient DQN algorithm for discounted reward Markov decision processes from Avrachenkov et al. (2021) to average reward problems. We experimentally compare widely used RVI Q-Learning with recently proposed Differential Q-Learning in the neural function approximation setting with Full Gradient DQN and DQN. We also extend this to learn Whittle indices for Markovian restless multi-armed bandits. We observe a better convergence rate of the proposed Full Gradient variant across different tasks.

Cited by

• MARBLE: Multi-Armed Restless Bandits in Latent Markovian Environment

Related

• Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme
• RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning
• Efficient Q-Learning and Actor-Critic Methods for Robust Average Reward Reinforcement Learning
• Using Deep Q-Learning to Control Optimization Hyperparameters
• A Differential Perspective on Distributional Reinforcement Learning