Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC

2024/11/06 by T. Alan Clark, Mark Towers, Clark, Tyler +5 · 3 citations
Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Green IT and Sustainability #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2411.03820

openalex publication_date 2024/11/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Rainbow Deep Q-Network (DQN) demonstrated combining multiple independent enhancements could significantly boost a reinforcement learning (RL) agent's performance. In this paper, we present "Beyond The Rainbow" (BTR), a novel algorithm that integrates six improvements from across the RL literature to Rainbow DQN, establishing a new state-of-the-art for RL using a desktop PC, with a human-normalized interquartile mean (IQM) of 7.4 on Atari-60. Beyond Atari, we demonstrate BTR's capability to handle complex 3D games, successfully training agents to play Super Mario Galaxy, Mario Kart, and Mortal Kombat with minimal algorithmic changes. Designing BTR with computational efficiency in mind, agents can be trained using a high-end desktop PC on 200 million Atari frames within 12 hours. Additionally, we conduct detailed ablation studies of each component, analyzing the performance and impact using numerous measures. Code is available at https://github.com/VIPTankz/BTR.

Cited by

• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Toward Agents That Reason About Their Computation
• A Comparative User Evaluation of XRL Explanations using Goal Identification

Related

• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep\n Reinforcement Learning Research
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Human-Level Control without Server-Grade Hardware
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Massively Parallel Methods for Deep Reinforcement Learning