Revisiting TD Target Aggregation under Uncertainty in Q-Learning

2026/08/04 by Lipeng Zu, Xiaonan Zhang
Computer Science · #cs.LG

paper · pdf

Accepted in the 42nd Conference on Uncertainty in Artificial Intelligence (UAI)

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the Successor Rollout Aggregation Deep Q-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.

Citations

• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning
• Simplifying Deep Temporal Difference Learning
• SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learning
• World Models via Policy-Guided Trajectory Diffusion
• RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability
• Efficient Dynamics Modeling in Interactive Environments with Koopman Theory
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
• Reinforcement Learning with Simple Sequence Priors
• Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning
• Synthetic Experience Replay
• Understanding Self-Predictive Learning for Reinforcement Learning
• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
• The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
• Denoised MDPs: Learning World Models Better Than the World Itself
• When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning
• Planning with Diffusion for Flexible Behavior Synthesis
• Revisiting Model-based Value Expansion
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Gradient Temporal-Difference Learning with Regularized Corrections
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning
• Target-Based Temporal Difference Learning
• CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification
• Learning Latent Dynamics for Planning from Pixels
• Recurrent World Models Facilitate Policy Evolution
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Efficient Exploration through Bayesian Deep Q-Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Distributional Reinforcement Learning with Quantile Regression
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning
• Deep Exploration via Bootstrapped DQN
• Asynchronous Methods for Deep Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Prioritized Experience Replay
• Deep Reinforcement Learning with Double Q-learning
• Human-level control through deep reinforcement learning
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Neuronlike adaptive elements that can solve difficult learning control problems
• Reinforcement Learning: An Introduction