Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow

2025/12/23 by Tyler Clark, Clark, Tyler, Christine A. Evers +3
Computer Science · #Reinforcement Learning in Robotics #Domain Adaptation and Few-Shot Learning #Advanced Neural Network Applications

paper · doi:10.48550/arxiv.2512.20513

Abstract

Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL algorithms, we observe a 35.6% human-normalized interquartile mean (IQM) performance improvement across the Atari benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework.

Citations

• Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC
• Simplifying Deep Temporal Difference Learning
• Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning
• Sovereign Byte Firewall: Comparing Transformer and Mamba-2 Backbones for Tokenization-Free Zero-Day Network Anomaly Detection
• Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
• EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data
• Recurrent Reinforcement Learning with Memoroids
• Vision-Language Models Provide Promptable Representations for Reinforcement Learning
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Reinforcement Learning with Fast and Forgetful Memory
• Small batch deep reinforcement learning
• Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
• Bigger, Better, Faster: Human-level Atari with human-level efficiency
• Structured State Space Models for In-Context Reinforcement Learning
• Mastering Diverse Domains through World Models
• Atari-5: Distilling the Arcade Learning Environment down to Five Games
• Human-level Atari 200x faster
• Fast and Data-Efficient Training of Rainbow: an Experimental Study on\n Atari
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Learning Transferable Visual Models From Natural Language Supervision
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep\n Reinforcement Learning Research
• Phasic Policy Gradient
• Agent57: Outperforming the Atari Human Benchmark
• Dota 2 with Large Scale Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Stabilizing Transformers for Reinforcement Learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Revisiting the Arcade Learning Environment: Evaluation Protocols and\n Open Problems for General Agents
• Proximal Policy Optimization Algorithms
• A Distributional Perspective on Reinforcement Learning
• Noisy Networks for Exploration
• Attention Is All You Need
• Safe and Efficient Off-Policy Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• Dueling Network Architectures for Deep Reinforcement Learning
• Prioritized Experience Replay
• Deep Reinforcement Learning with Double Q-learning
• Continuous control with deep reinforcement learning
• Effective Approaches to Attention-based Neural Machine Translation
• Deep Recurrent Q-Learning for Partially Observable MDPs
• Attention-Based Models for Speech Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Playing Atari with Deep Reinforcement Learning
• Gymnasium: A Standard Interface for Reinforcement Learning Environments

Related

• Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks
• How to Make Deep RL Work in Practice
• Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
• Real-Time Recurrent Learning using Trace Units in Reinforcement Learning
• ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models