Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
2026/06/04 by Johan Obando-Ceron, Lu Li, Scott Fujimoto +3 · 1 voice
#cs.LG #cs.AI
paper · pdf
Abstract
Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but representation learning. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, even without planning. We evaluate a simple model-free algorithm, MR.Q, coupled with auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.
Citations
- Learning Massively Multitask World Models for Continuous Control
- Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
- Training Agents Inside of Scalable World Models
- Video models are zero-shot learners and reasoners
- A Survey of State Representation Learning for Deep Reinforcement Learning
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
- The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer
- Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
- Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
- Hadamax Encoding: Elevating Performance in Model-Free Atari
- Towards General-Purpose Model-Free Reinforcement Learning
- CALE: Continuous Arcade Learning Environment
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Neuroplastic Expansion in Deep Reinforcement Learning
- Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
- ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
- PWM: Policy Learning with Multi-Task World Models
- On the consistency of hyper-parameter selection in value-based deep reinforcement learning
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
- In value-based deep reinforcement learning, a pruned network is a good network
- Mixtures of Experts Unlock Parameter Scaling for Deep RL
- Bridging State and History Representations: Understanding Self-Predictive RL
- TD-MPC2: Scalable, Robust World Models for Continuous Control
- Small batch deep reinforcement learning
- Simplified Temporal Consistency Reinforcement Learning
- For SALE: State-Action Representation Learning for Deep Reinforcement Learning
- Bigger, Better, Faster: Human-level Atari with human-level efficiency
- DINOv2: Learning Robust Visual Features without Supervision
- The Dormant Neuron Phenomenon in Deep Reinforcement Learning
- Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similarities
- On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
- Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective
- Large Language Models are Zero-Shot Reasoners
- The Primacy Bias in Deep Reinforcement Learning
- A Generalist Agent
- Flamingo: a Visual Language Model for Few-Shot Learning
- What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
- Value Gradient weighted Model-Based Reinforcement Learning
- Temporal Difference Learning for Model Predictive Control
- Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
- Procedural Generalization by Planning with Self-Supervised World Models
- Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
- MICo: Improved representations via sampling-based state similarity for Markov decision processes
- On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
- Learning Transferable Visual Models From Natural Language Supervision
- Multi-Task Reinforcement Learning with Context-based Representations
- Revisiting Rainbow: Promoting more Insightful and Inclusive Deep\n Reinforcement Learning Research
- Implicit Under-Parameterization Inhibits Data-Efficient Deep\n Reinforcement Learning
- Mastering Atari with Discrete World Models
- Predictive Information Accelerates Learning in RL
- Data-Efficient Reinforcement Learning with Self-Predictive\n Representations
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- CURL: Contrastive Unsupervised Representations for Reinforcement Learning
- Gradient Surgery for Multi-Task Learning
- Dream to Control: Learning Behaviors by Latent Imagination
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
- Solving Rubik's Cube with a Robot Hand
- When to Trust Your Model: Model-Based Policy Optimization
- DeepMDP: Learning Continuous Latent Space Models for Representation Learning
- Model-Based Reinforcement Learning for Atari
- Learning Latent Dynamics for Planning from Pixels
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- World Models
- Addressing Function Approximation Error in Actor-Critic Methods
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- DeepMind Control Suite
- Proximal Policy Optimization Algorithms
- Distral: Robust Multitask Reinforcement Learning
- Reinforcement Learning with Unsupervised Auxiliary Tasks
- EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
- Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
- Playing Atari with Deep Reinforcement Learning
Discussions
Related