Position: Benchmarking is Limited in Reinforcement Learning Research

2024/06/23 by Scott M. Jordan, Jordan, Scott M., Adam White +7 · 6 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Methodology (stat.ME) #Open Source Software Innovations

paper · pdf · doi:10.48550/arxiv.2406.16241

openalex publication_date 2024/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking.

Cited by

• gfnx: Fast and Scalable Library for Generative Flow Networks in JAX
• The Formalism-Implementation Gap in Reinforcement Learning Research
• Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviors
• Behavior-Consistent Deep Reinforcement Learning
• BenchQC -- Scalable and modular benchmarking of industrial quantum computing applications
• A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility

Related

• Can we hop in general? A discussion of benchmark selection and design using the Hopper environment
• Empirical Design in Reinforcement Learning
• Evaluating the Performance of Reinforcement Learning Algorithms
• The Formalism-Implementation Gap in Reinforcement Learning Research
• Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?