Rainbow: Combining Improvements in Deep Reinforcement Learning

2017/10/06 by Matteo Hessel, Joseph Modayil, Hessel, Matteo +17 · 88 citations
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1710.02298

openalex publication_date 2017/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Cited by

• Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms
• When Every Simulation Counts: Value-Based Reinforcement Learning for Accelerated Photonics Inverse Design
• Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning
• Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
• Trajectory Planning for UAV-Based Smart Farming Using Imitation-Based Triple Deep Q-Learning
• SACn: Soft Actor-Critic with n-step Returns
• Multi-Objective Reinforcement Learning for Large-Scale Mixed Traffic Control
• Analyzing Collision Rates in Large-Scale Mixed Traffic Control via Multi-Agent Reinforcement Learning
• How Market Volatility Shapes Algorithmic Collusion: A Comparative Analysis of Learning-Based Pricing Algorithms
• GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation
• Discovering Self-Protective Falling Policy for Humanoid Robot via Deep Reinforcement Learning
• RELiQ: Scalable Entanglement Routing via Reinforcement Learning in Quantum Networks
• Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning
• SpeedAug: Policy Acceleration via Tempo-Enriched Policy and RL Fine-Tuning
• Noise-Adaptive Quantum Circuit Mapping for Multi-Chip NISQ Systems via Deep Reinforcement Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
• Multi-Objective Adaptive Rate Limiting in Microservices Using Deep Reinforcement Learning
• Machine learning-based cloud resource allocation algorithms: a comprehensive comparative review
• Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
• D2 Actor Critic: Diffusion Actor Meets Distributional Critic
• NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation
• Learning Parameterized Skills from Demonstrations
• The Formalism-Implementation Gap in Reinforcement Learning Research
• Confounding Robust Deep Reinforcement Learning: A Causal Approach
• Enhancing Security in Deep Reinforcement Learning: A Comprehensive Survey on Adversarial Attacks and Defenses
• A Markov Decision Process for Variable Selection in Branch & Bound
• Actor-Free Continuous Control via Structurally Maximizable Q-Functions
• R2L: Reliable Reinforcement Learning: Guaranteed Return & Reliable Policies in Reinforcement Learning
• A Comparative User Evaluation of XRL Explanations using Goal Identification
• ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning
• DeepEN: A Deep Reinforcement Learning Framework for Personalized Enteral Nutrition in Critical Care
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions
• Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAX
• Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• DyMoDreamer: World Modeling with Dynamic Modulation
• Bridging Discrete and Continuous RL: Stable Deterministic Policy Gradient with Martingale Characterization
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• CrystalGym: A New Benchmark for Materials Discovery Using Reinforcement Learning
• Cascade Attribute Network: Decomposing Reinforcement Learning Control Policies using Hierarchical Neural Networks
• Activation Function Design Sustains Plasticity in Continual Learning
• Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning
• Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Affordance-based Reinforcement Learning for Urban Driving
• Approximating two value functions instead of one: towards characterizing\n a new family of Deep Reinforcement Learning algorithms
• Interference and Generalization in Temporal Difference Learning
• Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
• Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions
• Accelerating Atomic Fine Structure Determination with Graph Reinforcement Learning
• Training Larger Networks for Deep Reinforcement Learning
• Implicit Under-Parameterization Inhibits Data-Efficient Deep\n Reinforcement Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• WebSight: A Vision-First Architecture for Robust Web Agents
• Instance based Generalization in Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement\n Learning
• An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learning
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• A Comprehensive Review of Multi-Agent Reinforcement Learning in Video Games
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Adversary Agnostic Robust Deep Reinforcement Learning
• Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learning
• ReLATE: Learning Efficient Sparse Encoding for High-Performance Tensor Decomposition
• Breaking the Cold-Start Barrier: Reinforcement Learning with Double and Dueling DQNs
• Improving Generative Imagination in Object-Centric World Models
• Behavior From the Void: Unsupervised Active Pre-Training
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
• Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• Theory of Mind for Deep Reinforcement Learning in Hanabi
• Mixture of Step Returns in Bootstrapped DQN
• Mastering Rate based Curriculum Learning
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
• Natural Language-Driven Viewpoint Navigation for Volume Exploration via Semantic Block Representation
• QFOR: A Fidelity-aware Orchestrator for Quantum Computing Environments using Deep Reinforcement Learning
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• General non-linear Bellman equations
• CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning
• Bayesian Distributional Policy Gradients
• Is Exploration or Optimization the Problem for Deep Reinforcement Learning?
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling

Related

• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep\n Reinforcement Learning Research
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
• Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
• Faster Deep Reinforcement Learning with Slower Online Network
• Implementing the Deep Q-Network