Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

2024/03/09 by Marcel Hussing, Hussing, Marcel, Claas Voelcker +8 · 1 voice · 6 citations
Computer Science · Engineering · #Ferroelectric and Negative Capacitance Devices #Semiconductor materials and devices #VLSI and Analog Circuit Testing #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2403.05996

openalex publication_date 2024/03/09 · openalex created_date 2024/03/13 · openalex updated_date 2026/07/28

Abstract

We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples by combatting value function divergence. Under large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we investigate the phenomena leading to the primacy bias. We inspect the early stages of training that were conjectured to cause the failure to learn and find that one fundamental challenge is a long-standing acquaintance: value function divergence. Overinflated Q-values are found not only on out-of-distribution but also in-distribution data and can be linked to overestimation on unseen action prediction propelled by optimizer momentum. We employ a simple unit-ball normalization that enables learning under large update ratios, show its efficacy on the widely used dm_control suite, and obtain strong performance on the challenging dog tasks, competitive with model-based approaches. Our results question, in parts, the prior explanation for sub-optimal learning due to overfitting early data.

Cited by

• Relative Entropy Pathwise Policy Optimization
• Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits
• Learning Massively Multitask World Models for Continuous Control
• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

Discussions

• Been seeing a lot of RL architectures use layer norms as "design choice" without elaborating why it's needed. If you are curious take a look at our paper: arxiv.org/abs/2403.05996 TLDR: Layer norms st [bsky, 26 points, 1 comments]

Related

• The Primacy Bias in Deep Reinforcement Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting
• Loss of Plasticity in Continual Deep Reinforcement Learning
• Implicit Under-Parameterization Inhibits Data-Efficient Deep\n Reinforcement Learning