PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

2026/08/06 by Xi Zeng, Haojie Ren, Ziying Song
Computer Science · #cs.CV

paper · pdf

16 pages, 5 figures

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

We propose PhyLatent, a dynamics-relevant training objective for JointEmbedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not ensure that a representation preserves physical states and action consequences. We identify three failure modes in JEPA world models: physical invariance collapse, physical identifiability collapse, and counterfactual dynamics collapse. PhyLatent addresses them through three training pathways: physical invariance, physical identifiability, and counterfactual dynamics, implemented with physical state grounding, future representation alignment, static visual invariance, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three failure rates from 15.60%, 6.71%, and 8.41% to 7.53%, 0.95%, and 4.62%, respectively, and improves model predictive control (MPC) success from 70.0% to 78.1%. With the same architecture and planner, it further improves success from 81.0% to 98.0% on TwoRooms and remains competitive on Reacher and PushT. These results show that global non-collapse alone is insufficient for learning a reliable JEPA worldmodel state space.

Citations

• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Chimère Ω — blueprint for a physico-cognitively inspired local-first LLM runtime
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• OGBench: Benchmarking Offline Goal-Conditioned RL
• Revisiting Feature Prediction for Learning Visual Representations from Video
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• Denoised MDPs: Learning World Models Better Than the World Itself
• Masked Autoencoders Are Scalable Vision Learners
• BEiT: BERT Pre-Training of Image Transformers
• VICReg: Variance-Invariance-Covariance Regularization for\n Self-Supervised Learning
• Emerging Properties in Self-Supervised Vision Transformers
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Exploring Simple Siamese Representation Learning
• Exploring Simple Siamese Representation Learning
• dm_control: Software and Tasks for Continuous Control
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Bootstrap your own latent: A new approach to self-supervised Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Dream to Control: Learning Behaviors by Latent Imagination
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Learning Latent Dynamics for Planning from Pixels
• Representation Learning with Contrastive Predictive Coding
• Curiosity-driven Exploration by Self-supervised Prediction
• Time-Contrastive Networks: Self-Supervised Learning from Video