Overcoming Statistical Bias in Action-Controllable World Models

2026/08/05 by Yuhong Shi, Zhenhao Chu, Jie Wei +3
Computer Science · #cs.CV #cs.RO

paper · pdf

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Action-conditioned world models aim to predict how visual environments evolve under an agent's actions. Yet future frames are often highly predictable from visual inertia and recurring motion patterns alone. This creates a shortcut: models can fit the data by exploiting statistical biases without making their visible dynamics meaningfully depend on the action. As a result, different actions may produce similar futures, while motion may persist even under zero action. The key question is how to reduce reliance on statistical shortcuts from dominating action-conditioned prediction. We argue that action control requires more than injecting action features; it requires enforcing consistency under counterfactual changes to actions and observations. Based on this insight, we introduce CoCo, a Counterfactual Consistency framework to enhance action controllability through two complementary constraints. Multi-step counterfactual consistency constrains reference, inverse-action, and zero-action rollouts, while action-spatial counterfactual consistency enforces consistent predictions under mirrored scenes and transformed actions. Together, they reduce reliance on statistical shortcuts from substituting for action-dependent dynamics. We further introduce Action Response Consistency (ARC) and Drift Energy (DE) to assess action controllability, together with Mini-SSMB for same-state, multi-action counterfactual evaluation. On Mini-SSMB, our full model achieved ARC_inv of 0.412 and ARC_ref of 0.483, while reducing DE by 17.07% relative to the baseline. On VP2 visual planning, it achieves the highest average success rate among SOTA models, at 73.1%. Experiments on BAIR and RoboNet further show that these gains preserve video prediction quality and transfer across model settings.

Citations

• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion
• Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization
• What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
• RLVR-World: Training World Models with Reinforcement Learning
• Learning 3D Persistent Embodied World Models
• PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation
• Neural Motion Simulator: Pushing the Limit of World Models in Reinforcement Learning
• CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidance
• GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
• Multi-Task Interactive Robot Fleet Learning with Visual World Models
• Language Agents Meet Causality -- Bridging LLMs and Causal World Models
• Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement
• Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
• Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning
• iVideoGPT: Interactive VideoGPTs are Scalable World Models
• On the Content Bias in Fréchet Video Distance
• RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
• Robust agents learn causal world models
• Do Transformer World Models Give Better Policy Gradients?
• TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
• Finetuning Offline World Models in the Real World
• MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
• High Fidelity Image Counterfactuals with Probabilistic Causal Models
• Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks
• A Control-Centric Benchmark for Video Prediction
• MaskViT: Masked Visual Pre-Training for Video Prediction
• Counterfactual Analysis in Dynamic Latent State Models
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• On the Computation of PSNR for a Set of Images or Video
• RoboNet: Large-Scale Multi-Robot Learning
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Self-Supervised Visual Planning with Temporal Skip Connections
• FiLM: Visual Reasoning with a General Conditioning Layer
• Attention Is All You Need
• Image quality assessment: from error visibility to structural similarity