Simplifying Deep Temporal Difference Learning

2024/07/05 by Matteo Gallici, Gallici, Matteo, Mattie Fellows +15 · 2 voices · 40 citations
Computer Science · #Artificial intelligence #Computer science #Deep learning #Music and Audio Processing #Reinforcement learning #Speech Recognition and Synthesis #Speech and Audio Processing #Temporal difference learning #cs.LG

paper · pdf · doi:10.48550/arxiv.2407.04811

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Q-learning played a foundational role in the field reinforcement learning (RL). However, TD algorithms with off-policy data, such as Q-learning, or nonlinear function approximation like deep neural networks require several additional tricks to stabilise training, primarily a large replay buffer and target networks. Unfortunately, the delayed updating of frozen network parameters in the target network harms the sample efficiency and, similarly, the large replay buffer introduces memory and implementation overheads. In this paper, we investigate whether it is possible to accelerate and simplify off-policy TD training while maintaining its stability. Our key theoretical result demonstrates for the first time that regularisation techniques such as LayerNorm can yield provably convergent TD algorithms without the need for a target network or replay buffer, even with off-policy data. Empirically, we find that online, parallelised sampling enabled by vectorised environments stabilises training without the need for a large replay buffer. Motivated by these findings, we propose PQN, our simplified deep online Q-Learning algorithm. Surprisingly, this simple algorithm is competitive with more complex methods like: Rainbow in Atari, PPO-RNN in Craftax, QMix in Smax, and can be up to 50x faster than traditional DQN without sacrificing sample efficiency. In an era where PPO has become the go-to RL algorithm, PQN reestablishes off-policy Q-learning as a viable alternative.

Cited by

• Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning
• Replicable Reinforcement Learning with Linear Function Approximation
• Relative Entropy Pathwise Policy Optimization
• Hierarchical Reasoning Model
• Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments
• Value-Based Deep RL Scales Predictably
• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Understanding and Improving Hyperbolic Deep Reinforcement Learning
• Task diversity produces systematic transfer but inhibits continual reinforcement learning
• Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Training
• Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale
• Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
• A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Deep SPI: Safe Policy Improvement via World Models
• Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX
• Geometric Properties of Neural Multivariate Regression
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
• Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning
• Compute-Optimal Scaling for Value-Based Deep RL
• A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
• Gated Q-learning: Add Off-Policy Bias to Taste
• Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
• Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learning
• Horizon Reduction Makes RL Scalable
• The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
• FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control
• Universal Value-Function Uncertainties
• Hadamax Encoding: Elevating Performance in Model-Free Atari
• Scaling Multi Agent Reinforcement Learning for Underwater Acoustic Tracking via Autonomous Vehicles
• JaxRobotarium: Training and Deploying Multi-Robot Policies in 10 Minutes
• TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning
• Stable Deep Reinforcement Learning via Isotropic Gaussian Representations
• Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
• Revisiting TD Target Aggregation under Uncertainty in Q-Learning
• Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning

Discussions

• Super excited to share our paper, Simplifying Deep Temporal Difference Learning has been accepted as a spotlight at ICLR! My fab collaborator Matteo Gallici and I have written a three part blog on the [bsky, 20 points, 3 comments]
• 最終層直前にLayerNormを入れるのは、経験的にあまり良いことになった気がしないけどなぁ Simplifying Deep Temporal Difference Learning arxiv.org/abs/2407.04811 [bsky, 0 points, 0 comments]

Related

• Qgraph-bounded Q-learning: Stabilizing Model-Free Off-Policy Deep Reinforcement Learning
• Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning
• Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPs
• Making Deep Q-learning methods robust to time discretization
• Towards Characterizing Divergence in Deep Q-Learning