Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning

2023/09/22 by Lakshmi Narasimhan Govindarajan, Govindarajan, Lakshmi Narasimhan, Rex G. Liu +11
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Computer Vision and Pattern Recognition (cs.CV) #Digital Games and Media #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2309.13181

openalex publication_date 2023/09/22 · openalex created_date 2023/09/27 · openalex updated_date 2026/07/28

Abstract

Humans learn by interacting with their environments and perceiving the outcomes of their actions. A landmark in artificial intelligence has been the development of deep reinforcement learning (dRL) algorithms capable of doing the same in video games, on par with or better than humans. However, it remains unclear whether the successes of dRL models reflect advances in visual representation learning, the effectiveness of reinforcement learning algorithms at discovering better policies, or both. To address this question, we introduce the Learning Challenge Diagnosticator (LCD), a tool that separately measures the perceptual and reinforcement learning demands of a task. We use LCD to discover a novel taxonomy of challenges in the Procgen benchmark, and demonstrate that these predictions are both highly reliable and can instruct algorithmic development. More broadly, the LCD reveals multiple failure cases that can occur when optimizing dRL algorithms over entire video game benchmarks like Procgen, and provides a pathway towards more efficient progress.

Citations

• Adam: A Method for Stochastic Optimization
• CORTICAL ALGORITHMS FOR PERCEPTUAL GROUPING
• DeepMind Control Suite
• Evolved Policy Gradients
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Meta-Gradient Reinforcement Learning
• Learning Montezuma's Revenge from a Single Demonstration
• Go-Explore: a New Approach for Hard-Exploration Problems
• Generalisation in humans and deep neural networks
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Phasic Policy Gradient
• Long Range Arena: A Benchmark for Efficient Transformers
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Real-World Robot Learning with Masked Visual Pre-training
• Fully Convolutional Networks for Semantic Segmentation

Related

• Distilling Reinforcement Learning Tricks for Video Games
• Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• Transferring Deep Reinforcement Learning with Adversarial Objective and Augmentation
• Architecting and Visualizing Deep Reinforcement Learning Models