LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

2026/03/13 by Lucas Maes, Quentin Le Lidec, Damien Scieur +2 · 14 voices · 5 citations
#cs.LG #cs.AI

paper · pdf

Abstract

Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces, yet existing methods remain fragile, relying on complex multi-term losses, exponential moving averages, pre-trained encoders, or auxiliary supervision to avoid representation collapse. In this work, we introduce LeWorldModel (LeWM), the first JEPA that trains stably end-to-end from raw pixels using only two loss terms: a next-embedding prediction loss and a regularizer enforcing Gaussian-distributed latent embeddings. This reduces tunable loss hyperparameters from six to one compared to the only existing end-to-end alternative. With ~15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48x faster than foundation-model-based world models while remaining competitive across diverse 2D and 3D control tasks. Beyond control, we show that LeWM's latent space encodes meaningful physical structure through probing of physical quantities. Surprise evaluation confirms that the model reliably detects physically implausible events.

Citations

• stable-pretraining-v1: Foundation Model Research Made Simple
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime
• Training Agents Inside of Scalable World Models
• HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
• AI-Generated Video Detection via Perceptual Straightening
• Dual Perspectives on Non-Contrastive Self-Supervised Learning
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation
• Intuitive physics understanding emerges from self-supervised pretraining on natural videos
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
• OGBench: Benchmarking Offline Goal-Conditioned RL
• Brain-JEPA: Brain Dynamics Foundation Model with Gradient Positioning and Spatiotemporal Masking
• Efficient World Models with Context-Aware Tokenization
• Diffusion for World Modeling: Visual Details Matter in Atari
• TD-MPC2: Scalable, Robust World Models for Continuous Control
• DINOv2: Learning Robust Visual Features without Supervision
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• Mastering Diverse Domains through World Models
• Scalable Diffusion Models with Transformers
• Transformers are Sample-Efficient World Models
• Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
• Temporal Difference Learning for Model Predictive Control
• Offline Reinforcement Learning with Implicit Q-Learning
• VICReg: Variance-Invariance-Covariance Regularization for\n Self-Supervised Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Mastering Atari with Discrete World Models
• Dream to Control: Learning Behaviors by Latent Imagination
• Recurrent World Models Facilitate Policy Evolution
• World Models
• DeepMind Control Suite
• Layer Normalization
• End-to-End Training of Deep Visuomotor Policies
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gymnasium: A Standard Interface for Reinforcement Learning Environments

Cited by

• ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games
• Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method
• A Comprehensive Survey on World Models for Embodied AI
• ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow
• What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations
• QQWorld: Quantile-Quantile Matching for World Model Regularization

Discussions

• LeWorldModel: Stable End-to-End Predictive Architecture from Pixels [hn, 6 points, 0 comments]
• Stable End-to-End Joint-Embedding Predictive Architecture from Pixels [hn, 5 points, 0 comments]
• LeWorldModel with Yann LeCun [hn, 2 points, 1 comments]
• Aquí lo tienes arxiv.org/abs/2603.19312 [bsky, 2 points, 2 comments]
• LeWorldModel: Stable E2E Joint-Embedding Predictive Architecture from Pixels [hn, 2 points, 0 comments]
• New breakthrough in World Models allows the AI to actually learn the underlying physics to make vision predictions with a tiny 15M param model, trainable on a single GPU arxiv.org/pdf/2603.19312 [bsky, 1 points, 1 comments]
• Paper review: LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels https://arxiv.org/pdf/2603.19312 Nice clean github: https://github.com/lucas-maes/le-wm (1/14) [bsky, 0 points, 1 comments]
• arxiv.org/abs/2603.19312 [bsky, 0 points, 0 comments]
• [2603.19312v1] LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — A clear look at LeWorldModel, an end-to-end joint-embedding predictive setup that learns a stable w [bsky, 0 points, 0 comments]
• 今日読んだ論文 LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels arxiv.org/abs/2603.19312 JEPA的なやり方で表現空間における次状態予測損失で世界モデルを学習する。stop-gradientやEMAなし、次状態予測+SIGRegだけで学習に成功しているのは [bsky, 0 points, 0 comments]
• LeWorldModel strengthens Yann LeCun’s long standing argument that latent predictive world models may be far more efficient than brute force generative approaches for learning actionable structure from [bsky, 0 points, 0 comments]
• 30:26 - Isaac Ward — LeWorldModeling (https://arxiv.org/abs/2603.19312) 43:54 - Akshay Vegesna (@akshayvegesna) — Deep Learning is Not So Mysterious or Different (https://arxiv.org/abs/2503.02113) 51: [bsky, 0 points, 1 comments]
• Interesting references: arxiv.org/abs/2603.19312 [bsky, 0 points, 1 comments]
• ちなみにこれ。 arxiv.org/abs/2603.19312 [bsky, 0 points, 0 comments]

Related

• Joint Embedding Predictive Architectures Focus on Slow Features
• Learning and Leveraging World Models in Visual Representation Learning
• Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control
• CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Images
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime