Bigger, Better, Faster: Human-level Atari with human-level efficiency

2023/05/30 by Max Schwarzer, Schwarzer, Max, Johan Obando-Ceron +10 · 1 voice · 27 citations
Computer Science · #Anomaly Detection Techniques and Applications #Human Pose and Action Recognition #Reinforcement Learning in Robotics #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2305.19452

openalex publication_date 2023/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this scaling in a sample-efficient manner. We conduct extensive analyses of these design choices and provide insights for future work. We end with a discussion about updating the goalposts for sample-efficient RL research on the ALE. We make our code and data publicly available at https://github.com/google-research/google-research/tree/master/bigger_better_faster.

Cited by

• Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
• Relative Entropy Pathwise Policy Optimization
• Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
• Value-Based Deep RL Scales Predictably
• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Learning Sim-to-Real Humanoid Locomotion in 15 Minutes
• Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
• Natural Building Blocks for Structured World Models: Theory, Evidence, and Scaling
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach
• The Formalism-Implementation Gap in Reinforcement Learning Research
• Guardian: Decoupling Exploration from Safety in Reinforcement Learning
• Toward Agents That Reason About Their Computation
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• DyMoDreamer: World Modeling with Dynamic Modulation
• ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse
• Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation
• On the Limits of Tabular Hardness Metrics for Deep RL: A Study with the Pharos Benchmark
• Compute-Optimal Scaling for Value-Based Deep RL
• Sample-efficient LLM Optimization with Reset Replay
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
• What Can Grokking Teach Us About Learning Under Nonstationarity?

Discussions

• Last year I gave a talk titled "From 'Bigger, Better, Faster' to 'Smaller, Sparser, Stranger'", which looked at the components that make up our BBF agent (arxiv.org/abs/2305.19452), highlighting some [bsky, 67 points, 3 comments]

Related

• Human-level Atari 200x faster
• A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions
• Agent57: Outperforming the Atari Human Benchmark
• Mastering Atari Games with Limited Data
• Observe and Look Further: Achieving Consistent Performance on Atari