Learning Massively Multitask World Models for Continuous Control

2025/11/24 by Nicklas Hansen, Hao Su, Hansen, Nicklas +3 · 1 citation
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #Robot Manipulation and Learning #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2511.19584

openalex publication_date 2025/11/24 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

General-purpose control demands agents that act across many tasks and embodiments, yet research on reinforcement learning (RL) for continuous control remains dominated by single-task or offline regimes, reinforcing a view that online RL does not scale. Inspired by the foundation model recipe (large-scale pretraining followed by light RL) we ask whether a single agent can be trained on hundreds of tasks with online interaction. To accelerate research in this direction, we introduce a new benchmark with 200 diverse tasks spanning many domains and embodiments, each with language instructions, demonstrations, and optionally image observations. We then present Newt, a language-conditioned multitask world model that is first pretrained on demonstrations to acquire task-aware representations and action priors, and then jointly optimized with online interaction across all tasks. Experiments show that Newt yields better multitask performance and data-efficiency than a set of strong baselines, exhibits strong open-loop control, and enables rapid adaptation to unseen tasks. We release our environments, demonstrations, code for training and evaluation, as well as 200+ checkpoints.

Citations

• Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
• FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• High-Performance Reinforcement Learning on Spot: Optimizing Simulation Parameters with Distributional Measures
• Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
• TD-M(PC)²: Improving Temporal Difference MPC Through Policy Constraint
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks
• π₀: A Vision-Language-Action Flow Model for General Robot Control
• OGBench: Benchmarking Offline Goal-Conditioned RL
• ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
• A Super-human Vision-based Reinforcement Learning Agent for Autonomous Racing in Gran Turismo
• Octo: An Open-Source Generalist Robot Policy
• BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
• Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Finetuning Offline World Models in the Real World
• Extreme Parkour with Legged Robots
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
• A Generalist Dynamics Model for Control
• DINOv2: Learning Robust Visual Features without Supervision
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Diffusion policy: Visuomotor policy learning via action diffusion
• Efficient Online Reinforcement Learning with Offline Data
• Mastering Diverse Domains through World Models
• On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline
• MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
• Skill-based Model-based Reinforcement Learning
• Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
• EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine
• Multi-Game Decision Transformers
• A Generalist Agent
• Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
• Temporal Difference Learning for Model Predictive Control
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Bootstrap your own latent: A new approach to self-supervised Learning
• Planning to Explore via Self-Supervised World Models
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Mastering Atari, Go, chess and shogi by planning with a learned model
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Mish: A Self Regularized Non-Monotonic Activation Function
• Habitat: A Platform for Embodied AI Research
• Learning Latent Dynamics for Planning from Pixels
• Addressing Function Approximation Error in Actor-Critic Methods
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Neural Discrete Representation Learning
• Learning Complex Dexterous Manipulation with Deep Reinforcement Learning\n and Demonstrations
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Proximal Policy Optimization Algorithms
• A Distributional Perspective on Reinforcement Learning
• Attention Is All You Need
• Curiosity-driven Exploration by Self-supervised Prediction
• One-Shot Imitation Learning
• Robust Adversarial Reinforcement Learning
• Layer Normalization
• Generative Adversarial Imitation Learning
• Unifying Count-Based Exploration and Intrinsic Motivation
• Continuous control with deep reinforcement learning
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• A Reduction of Imitation Learning and Structured Prediction to No-Regret\n Online Learning
• Planning and acting in partially observable stochastic domains
• Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

Cited by

• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

Related

• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Multi-task Learning for Continuous Control
• PWM: Policy Learning with Multi-Task World Models
• On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
• Multi-Task Reinforcement Learning in Continuous Control with Successor Feature-Based Concurrent Composition