The Formalism-Implementation Gap in Reinforcement Learning Research

2025/10/17 by Pablo Samuel Castro, Castro, Pablo Samuel · 2 voices
Computer Science · Engineering · Neuroscience · #Embodied and Extended Cognition #Reinforcement Learning in Robotics #Robot Manipulation and Learning #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2510.16175

openalex publication_date 2025/10/17 · openalex created_date 2025/10/22 · openalex updated_date 2026/07/28

Abstract

The last decade has seen an upswing in interest and adoption of reinforcement learning (RL) techniques, in large part due to its demonstrated capabilities at performing certain tasks at "super-human levels". This has incentivized the community to prioritize research that demonstrates RL agent performance, often at the expense of research aimed at understanding their learning dynamics. Performance-focused research runs the risk of overfitting on academic benchmarks -- thereby rendering them less useful -- which can make it difficult to transfer proposed techniques to novel problems. Further, it implicitly diminishes work that does not push the performance-frontier, but aims at improving our understanding of these techniques. This paper argues two points: (i) RL research should stop focusing solely on demonstrating agent capabilities, and focus more on advancing the science and understanding of reinforcement learning; and (ii) we need to be more precise on how our benchmarks map to the underlying mathematical formalisms. We use the popular Arcade Learning Environment (ALE; Bellemare et al., 2013) as an example of a benchmark that, despite being increasingly considered "saturated", can be effectively used for developing this understanding, and facilitating the deployment of RL techniques in impactful real-world problems.

Citations

• A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
• CALE: Continuous Arcade Learning Environment
• Three Dogmas of Reinforcement Learning
• On the consistency of hyper-parameter selection in value-based deep reinforcement learning
• Position: Benchmarking is Limited in Reinforcement Learning Research
• Position: Why We Must Rethink Empirical Research in Machine Learning
• A Better Match for Drivers and Riders: Reinforcement Learning at Lyft
• Small batch deep reinforcement learning
• OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks
• Empirical Design in Reinforcement Learning
• Atari-5: Distilling the Arcade Learning Environment down to Five Games
• Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversions
• The State of Sparse Training in Deep Reinforcement Learning
• Mask Atari for Deep Reinforcement Learning as POMDP Benchmarks
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Reinforcement Learning Agents for Ubisoft's Roller Champions
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep\n Reinforcement Learning Research
• Multiplayer Support for the Arcade Learning Environment
• Discounted Reinforcement Learning Is Not an Optimization Problem
• Model-Based Reinforcement Learning for Atari
• Hyperbolic Discounting and Learning over Multiple Horizons
• Separating value functions across time-scales
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Generalization and Regularization in DQN
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Revisiting the Arcade Learning Environment: Evaluation Protocols and\n Open Problems for General Agents
• Planning and acting in partially observable stochastic domains
• Gymnasium: A Standard Interface for Reinforcement Learning Environments

Discussions

• This is something I talk about in my paper, where I suggest being explicit about {\gamma}_train (some methods use multiple gammas during training) and \gamma_eval. One of my students is empirically in [bsky, 12 points, 1 comments]
• Added to the reading list arxiv.org/abs/2510.16175 [bsky, 1 points, 2 comments]

Related

• A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
• Reinforcement Learning in Practice: Opportunities and Challenges
• Deep Reinforcement Learning Agents are not even close to Human Intelligence