Genie: Generative Interactive Environments

2024/02/23 by Jake Bruce, Michael J. Dennis, Bruce, Jake +50 · 11 voices · 120 citations
Social Sciences · #Digital Games and Media

paper · pdf · doi:10.48550/arxiv.2402.15391

Abstract

We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

Cited by

• KAN-Dreamer: Benchmarking Kolmogorov-Arnold Networks as Function Approximators in World Models
• LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
• Being-H0.7: A Latent World-Action Model from Egocentric Videos
• N₀-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation
• WCM: World-Cognition Model for Generalizable Human-Robot Interaction
• PhysFire-WM: A Physics-Informed World Model for Emulating Fire Spread Dynamics
• AstraNav-World: World Model for Foresight Control and Consistency
• Learning Skills from Action-Free Videos
• SemanticGen: Video Generation in Semantic Space
• 4D Gaussian Splatting as a Learned Dynamical System
• Dexterous World Models
• Spectral Representation-based Reinforcement Learning
• Spatia: Video Generation with Updatable Spatial Memory
• MobileWorldBench: Towards Semantic World Modeling For Mobile Agents
• World Models Can Leverage Human Videos for Dexterous Manipulation
• Motus: A Unified Latent Action World Model
• Flowception: Temporally Expansive Flow Matching for Video Generation
• WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control
• Evaluating Gemini Robotics Policies in a Veo World Simulator
• AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
• Astra: General Interactive World Model with Autoregressive Denoising
• CLARITY: Medical World Model for Guiding Treatment Decisions by Modeling Context-Aware Disease Trajectories in Latent Space
• See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
• TV2TV: A Unified Framework for Interleaved Language and Video Generation
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
• BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models
• RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL
• AdaPower: Specializing World Foundation Models for Predictive Manipulation
• GeoVideo: Introducing Geometric Regularization into Video Generation Model
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling
• Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
• From monoliths to modules: Decomposing transducers for efficient world modelling
• Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now
• IGen: Scalable Data Generation for Robot Learning from Open-World Images
• What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards
• SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments
• LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models
• Reinforcing Action Policies by Prophesying
• Thinking in 360°: Humanoid Visual Search in the Wild
• Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis
• Thinking Ahead: Foresight Intelligence in MLLMs and World Models
• Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?
• RynnVLA-002: A Unified Vision-Language-Action and World Model
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion
• LAOF: Robust Latent Action Learning with Optical Flow Constraints
• IPR-1: Interactive Physical Reasoner
• Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark
• Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos
• TempoMaster: Efficient Long Video Generation via Next-Frame-Rate Prediction
• Scalable Policy Evaluation with Video World Models
• Simulating the Visual World with Artificial Intelligence: A Roadmap
• MotionStream: Real-Time Video Generation with Interactive Motion Controls
• Next-Latent Prediction Transformers Learn Compact World Models
• WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models
• A Step Toward World Models: A Survey on Robotic Manipulation
• Co-Evolving Latent Action World Models
• Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models
• Enfold: Folding World-Generator Computation into Predictive Representations for Efficient Embodied Control
• Nautilus: From One Prompt to Plug-and-Play Robot Learning
• Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
• What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?
• Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents
• Semantic Communications with World Models
• Evaluating Video Models as Simulators of Multi-Person Pedestrian Trajectories
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
• A Comprehensive Survey on World Models for Embodied AI
• Terra: Explorable Native 3D World Model with Point Latents
• CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas
• Counting Hallucinations in Diffusion Models
• Ego-Vision World Model for Humanoid Contact Planning
• Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
• HiMaCon: Discovering Hierarchical Manipulation Concepts from Unlabeled Multi-Modal Data
• SoundReactor: Frame-level Online Video-to-Audio Generation
• DreamX-World 1.0: A General-Purpose Interactive World Model
• Ctrl-VI: Controllable Video Synthesis via Variational Inference
• Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report
• StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation
• Scaling Sequence-to-Sequence Generative Neural Rendering
• What Drives Compositional Generalization in Visual Generative Models?
• On the Identifiability of Latent Action Policies
• Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
• Seeing Space and Motion: Enhancing Latent Actions with Spatial and Dynamic Awareness for VLA
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
• NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding
• WoW: Towards a World omniscient World model Through Embodied Interaction
• EgoDemoGen: Novel Egocentric Demonstration Generation Enables Viewpoint-Robust Manipulation
• LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE
• X-Streamer: Unified Human World Modeling with Audiovisual Interaction
• Embodied AI: From LLMs to World Models
• ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow
• World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
• Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
• ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow
• World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
• How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective
• Latent Action Pretraining Through World Modeling
• Remote Sensing-Oriented World Model
• Prepare Before You Act: Learning From Humans to Rearrange Initial States
• SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world models
• WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
• Enhancing Physical Consistency in Lightweight World Models
• Imagined Autocurricula
• COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalization
• Planning with Reasoning using Vision Language World Model
• Learning Primitive Embodied World Models: Towards Scalable Robotic Learning
• WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception
• Precise Action-to-Video Generation Through Visual Action Prompts
• Matrix-game 2.0: An open-source real-time and streaming interactive world model
• Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search
• Large Model Empowered Embodied AI: A Survey on Decision-Making and Embodied Learning
• Yan: Foundational Interactive Video Generation
• Boosting Action-Information via a Variational Bottleneck on Unlabelled Robot Videos
• GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
• Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
• Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
• RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models
• villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

Discussions

• Ein Forschungspapier von Google DeepMind zu Genie. Vermutlich sind es gerade diese Bilder, die den Eindruck vermitteln, es handele sich um einen Game‑Generator. Wobei das durchaus ein Nebeneffekt sein [bsky, 7 points, 1 comments]
• Genie: Generative Interactive Environments [hn, 5 points, 1 comments]
• Genie: Generative Interactive Environments [hn, 3 points, 0 comments]
• Genie: Generative Interactive Environments [hn, 2 points, 1 comments]
• Genie did this in a really cool manner: arxiv.org/abs/2402.15391 [bsky, 2 points, 0 comments]
• Genie: Generative Interactive Environments [hn, 2 points, 0 comments]
• Genie: Generative Interactive Environments [hn, 1 points, 0 comments]
• Oh, actually, Genie 1 was a 2D WM arxiv.org/pdf/2402.15391 [bsky, 0 points, 0 comments]
• I'm reading a paper from Google DeepMind about Genie, which won an award at ICML2024! The technology is fascinating, but does it overwhelm game creators? I will post my blog about it. arxiv.org/abs [bsky, 0 points, 0 comments]
• Genie: Generative Interactive Environments - arxiv.org/abs/2402.15391 "can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, ph [bsky, 0 points, 0 comments]
• arxiv.org/abs/2402.15391 [bsky, 0 points, 0 comments]

Related

• Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
• Pandora: Towards General World Model with Natural Language Actions and Video States
• Exploration-Driven Generative Interactive Environments
• Learning Interactive Real-World Simulators
• VideoWorld: Exploring Knowledge Learning from Unlabeled Videos