Correcting Momentum in Temporal Difference Learning

2021/06/07 by Emmanuel Bengio, Joëlle Pineau, Bengio, Emmanuel +4 · 2 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2106.03955

NeurIPS Deep RL Workshop 2020

arxiv created 2021/06/07 · openalex publication_date 2021/06/07 · arxiv updated 2021/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale: not only does the gradient of the loss change due to parameter updates, the loss itself changes due to bootstrapping. We first show that this phenomenon exists, and then propose a first-order correction term to momentum. We show that this correction term improves sample efficiency in policy evaluation by correcting target value drift. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.

Citations

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• A Markovian Decision Process
• Accelerated Gradient Temporal Difference Learning
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Generalization and Regularization in DQN
• Assessing Generalization in Deep Reinforcement Learning
• Towards Characterizing Divergence in Deep Q-Learning
• Deep Residual Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?

Cited by

• ANO : Faster is Better in Noisy Landscape
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning

Related

• Gradient Temporal Difference with Momentum: Stability and Convergence
• Momentum in Reinforcement Learning
• The Marginal Value of Momentum for Small Learning Rate SGD
• Towards understanding how momentum improves generalization in deep learning
• When and Why Momentum Accelerates SGD:An Empirical Study