Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

2018/09/15 by Prabhat Nagarajan, Nagarajan, Prabhat, Garrett Warnell +3 · 10 citations
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning and Data Classification #Reinforcement Learning in Robotics #Software Engineering Research #cs.AI

paper · pdf · doi:10.48550/arxiv.1809.05676

17 Pages

openalex publication_date 2018/09/15 · arxiv created 2019/06/09 · arxiv updated 2019/06/11 · openalex created_date 2022/08/03 · openalex updated_date 2026/07/28

Abstract

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminism in the training process, which can substantially affect the results. Motivated by this challenge, we study the positive impacts of deterministic implementations in eliminating nondeterminism in training. To do so, we consider the particular case of the deep Q-learning algorithm, for which we produce a deterministic implementation by identifying and controlling all sources of nondeterminism in the training process. One by one, we then allow individual sources of nondeterminism to affect our otherwise deterministic implementation, and measure the impact of each source on the variance in performance. We find that individual sources of nondeterminism can substantially impact the performance of agent, illustrating the benefits of deterministic implementations. In addition, we also discuss the important role of deterministic implementations in achieving exact replicability of results.

Citations

• DeepMind Control Suite
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

Cited by

• Hyperparameter Optimization and Reproducibility in Deep Learning Model Training
• Machine learning for combinatorial optimization: A methodological tour d’horizon
• MuFlex: A Scalable, Physics-based Platform for Multi-Building Flexibility Analysis and Coordination
• UR²: Unify RAG and Reasoning through Reinforcement Learning
• Anti-Distillation: Improving reproducibility of deep networks
• The Scientific Method in the Science of Machine Learning
• Synthesizing Irreproducibility in Deep Networks
• Reproducibility in machine‐learning‐based research: Overview, barriers, and drivers
• Measuring the Reliability of Reinforcement Learning Algorithms
• Omega-S: A Functional Resilience Index for LLM Fine-Tuning

Related

• Making Deep Q-learning methods robust to time discretization
• Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
• Deep Reinforcement Learning that Matters
• Towards Characterizing Divergence in Deep Q-Learning
• Deep Q-Learning: Theoretical Insights from an Asymptotic Analysis