Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments

2019/04/12 by Kaleigh Clary, Emma Tosch, Clary, Kaleigh +5 · 2 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1904.06312

openalex publication_date 2019/04/12 · openalex created_date 2019/05/03 · openalex updated_date 2026/07/28

Abstract

Reproducibility in reinforcement learning is challenging: uncontrolled stochasticity from many sources, such as the learning algorithm, the learned policy, and the environment itself have led researchers to report the performance of learned agents using aggregate metrics of performance over multiple random seeds for a single environment. Unfortunately, there are still pernicious sources of variability in reinforcement learning agents that make reporting common summary statistics an unsound metric for performance. Our experiments demonstrate the variability of common agents used in the popular OpenAI Baselines repository. We make the case for reporting post-training agent performance as a distribution, rather than a point estimate.

Citations

• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Cited by

• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• RAPTOR: A Foundation Policy for Quadrotor Control

Related

• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Assessing the Impact of Distribution Shift on Reinforcement Learning Performance
• Deep Reinforcement Learning that Matters
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari