Small batch deep reinforcement learning

2023/10/05 by Johan Obando-Ceron, Marc G. Bellemare, Obando-Ceron, Johan +3 · 5 citations
Engineering · Neuroscience · #Artificial Intelligence (cs.AI) #EEG and Brain-Computer Interfaces #FOS: Computer and information sciences #Machine Learning (cs.LG) #Muscle activation and electromyography studies

paper · pdf · doi:10.48550/arxiv.2310.03882

openalex publication_date 2023/10/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In value-based deep reinforcement learning with replay memories, the batch size parameter specifies how many transitions to sample for each gradient update. Although critical to the learning process, this value is typically not adjusted when proposing new algorithms. In this work we present a broad empirical study that suggests \em reducing the batch size can result in a number of significant performance gains; this is surprising, as the general tendency when training neural networks is towards larger batch sizes for improved performance. We complement our experimental findings with a set of empirical analyses towards better understanding this phenomenon.

Cited by

• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• The Formalism-Implementation Gap in Reinforcement Learning Research
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Compute-Optimal Scaling for Value-Based Deep RL

Related

• An Empirical Model of Large-Batch Training
• Revisiting Small Batch Training for Deep Neural Networks
• Small Batch Sizes Improve Training of Low-Resource Neural MT
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model