Compute-Optimal Scaling for Value-Based Deep RL

2025/08/20 by Fu, Preston, Rybkin, Oleh, Zhou, Zhiyuan +4 · 2 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.14881

Abstract

As models grow larger and training them becomes expensive, it becomes increasingly important to scale training recipes not just to larger models and more data, but to do so in a compute-optimal manner that extracts maximal performance per unit of compute. While such scaling has been well studied for language modeling, reinforcement learning (RL) has received less attention in this regard. In this paper, we investigate compute scaling for online, value-based deep RL. These methods present two primary axes for compute allocation: model capacity and the update-to-data (UTD) ratio. Given a fixed compute budget, we ask: how should resources be partitioned across these axes to maximize sample efficiency? Our analysis reveals a nuanced interplay between model size, batch size, and UTD. In particular, we identify a phenomenon we call TD-overfitting: increasing the batch quickly harms Q-function accuracy for small models, but this effect is absent in large models, enabling effective use of large batch size at scale. We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD to optimize compute usage. Our findings provide a grounded starting point for compute-optimal scaling in deep RL, mirroring studies in supervised learning but adapted to TD learning.

Citations

• A Clean Slate for Offline Reinforcement Learning
• Scaling Laws for Native Multimodal Models
• 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning
• Value-Based Deep RL Scales Predictably
• SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Llama 3 Herd of Models
• SAPG: Split and Aggregate Policy Gradients
• Simplifying Deep Temporal Difference Learning
• Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
• HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation
• Language models scale reliably with over-training and on downstream tasks
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
• Scaling Laws for Fine-Grained Mixture of Experts
• Offline Actor-Critic Reinforcement Learning Scales to Large Models
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Small batch deep reinforcement learning
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• Scaling Data-Constrained Language Models
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting
• Understanding plasticity in neural networks
• Scaling Vision Transformers to 22 Billion Parameters
• Scaling laws for single-agent reinforcement learning
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
• Scaling Laws for Reward Model Overoptimization
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• The Primacy Bias in Deep Reinforcement Learning
• Training Compute-Optimal Large Language Models
• Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
• DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Scaling Vision Transformers
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Implicit Under-Parameterization Inhibits Data-Efficient Deep\n Reinforcement Learning
• Scaling Laws for Neural Language Models
• Mastering Atari, Go, chess and shogi by planning with a learned model
• An Empirical Model of Large-Batch Training
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• DeepMind Control Suite
• Asynchronous Methods for Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Cited by

• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
• floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL

Related

• The Art of Scaling Reinforcement Learning Compute for LLMs
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
• Value-Based Deep RL Scales Predictably
• More Compute Is What You Need
• Resolving Discrepancies in Compute-Optimal Scaling of Language Models