Transformers are Sample-Efficient World Models

2022/09/01 by Vincent Micheli, Eloi Alonso, Micheli, Vincent +3 · 2 voices · 53 citations
Computer Science · #Data Stream Mining Techniques #Machine Learning and Data Classification #Reinforcement Learning in Robotics #cs.AI #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2209.00588

openalex publication_date 2022/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-efficient reinforcement learning, we release our code and models at https://github.com/eloialonso/iris.

Cited by

• DWM: Separating World Effects from Actions in Latent World Models
• Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agents
• Flow Matching in Feature Space for Stochastic World Modeling
• Qwen-AgentWorld: Language World Models for General Agents
• Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models
• Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Dynamics Models
• GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation
• Spectral Representation-based Reinforcement Learning
• Closing the Train-Test Gap in World Models for Gradient-Based Planning
• Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now
• SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments
• RynnVLA-002: A Unified Vision-Language-Action and World Model
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion
• X-WIN: Building Chest Radiograph World Model via Predictive Sensing
• Object-Centric World Models for Causality-Aware Reinforcement Learning
• Simulating the Visual World with Artificial Intelligence: A Roadmap
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning
• When Object-Centric World Models Meet Policy Learning: From Pixels to Policies, and Where It Breaks
• Bootstrap Off-policy with World Model
• Human Machine Social Hybrid Intelligence:A Collaborative Decision Making Framework for Large Model Agent Groups and Human Experts
• WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
• World Models Should Prioritize the Unification of Physical and Social Dynamics
• Social World Model-Augmented Mechanism Design Policy Learning
• Socialized Learning and Emergent Behaviors in Multi-Agent Systems based on Multimodal Large Language Models
• Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
• A Comprehensive Survey on World Models for Embodied AI
• One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report
• MultiModal Action Conditioned Video Generation
• SITCOM: Scaling Inference-Time COMpute for VLAs
• DyMoDreamer: World Modeling with Dynamic Modulation
• Training Agents Inside of Scalable World Models
• PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Control
• In-Context Compositional Q-Learning for Offline Reinforcement Learning
• Adversarial Diffusion for Robust Reinforcement Learning
• Model-Based Reinforcement Learning under Random Observation Delays
• Embodied AI: From LLMs to World Models
• DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
• Temporal Straightening for Latent Planning
• EigenSafe: A Spectral Framework for Learning-Based Probabilistic Safety Assessment
• GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
• TransZero: Parallel Tree Expansion in MuZero using Transformer Networks
• One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
• Constrained Decoding for Robotics Foundation Models
• STORI: A Benchmark and Taxonomy for Stochastic Environments
• Learning Lifted Action Models From Traces of Incomplete Actions and States
• Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
• Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
• How Safe Will I Be Given What I Saw? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy
• Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
• In-Context Reinforcement Learning via Communicative World Models
• Back to the Features: DINO as a Foundation for Video World Models

Discussions

• Transformers are sample-efficient world models [hn, 67 points, 10 comments]
• Transformers Are Sample Efficient World Models [hn, 7 points, 0 comments]

Related

• Learning to Play Atari in a World of Tokens
• Transformer-based World Models Are Happy With 100k Interactions
• STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning
• Efficient World Models with Context-Aware Tokenization
• Transformers in Reinforcement Learning: A Survey