Diagnosing Bottlenecks in Deep Q-learning Algorithms

2019/02/26 by Justin Fu, Fu, Justin, Aviral Kumar +5 · 9 citations
Computer Science · Engineering · #Advancements in Semiconductor Devices and Circuit Design #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1902.10250

openalex publication_date 2019/02/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods. We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains.

Cited by

• Geometric Value Iteration: Dynamic Error-Aware KL Regularization for Reinforcement Learning
• On the Estimation Bias in Double Q-Learning
• Behavior Regularized Offline Reinforcement Learning
• A New Approach for Tactical Decision Making in Lane Changing: Sample\n Efficient Deep Q Learning with a Safety Feedback Reward
• Value Flows
• Training Larger Networks for Deep Reinforcement Learning
• DROMO: Distributionally Robust Offline Model-based Policy Optimization
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models

Related

• Towards Characterizing Divergence in Deep Q-Learning
• Deep Q-Learning: Theoretical Insights from an Asymptotic Analysis
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning
• Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning
• Is Q-learning an Ill-posed Problem?