Evaluating the World Model Implicit in a Generative Model

2024/06/06 by Keyon Vafa, Vafa, Keyon, Justin Y. Chen +7 · 18 voices · 25 citations
#cs.CL #cs.AI

paper · pdf · doi:10.48550/arxiv.2406.03689

Abstract

Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton. This includes problems as diverse as simple logical reasoning, geographic navigation, game-playing, and chemistry. We propose new evaluation metrics for world model recovery inspired by the classic Myhill-Nerode theorem from language theory. We illustrate their utility in three domains: game playing, logic puzzles, and navigation. In all domains, the generative models we consider do well on existing diagnostics for assessing world models, but our evaluation metrics reveal their world models to be far less coherent than they appear. Such incoherence creates fragility: using a generative model to solve related but subtly different tasks can lead to failures. Building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable; our results suggest new ways to assess how close a given model is to that goal.

Cited by

• Machine understanding
• The Impossibility of Inverse Permutation Learning in Transformer Models
• If generative AI is the answer, what is the question?
• Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
• Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
• Potemkin Understanding in Large Language Models
• Large Language Models and Emergence: A Complex Systems Perspective
• Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
• Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
• Cognitive Dark Matter: Measuring What AI Misses
• Generative Actor Critic
• AI Needs Physics More Than Physics Needs AI
• Agency at the Interface: Distinguishing Teleological from Structural Self-Organization via Internal Coarse-Graining and Downward Causation
• PragWorld: A Benchmark Evaluating LLMs' Local World Model under Minimal Linguistic Alterations and Conversational Dynamics
• MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddings
• AIA Forecaster: Technical Report
• Next-Latent Prediction Transformers Learn Compact World Models
• One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration
• CORE: Full-Path Evaluation of LLM Agents Beyond Final State
• From Next Token Prediction to (STRIPS) World Models -- Preliminary Results
• Stability-Aware Joint Communication and Control for Nonlinear Control-Non-Affine Wireless Networked Control Systems
• Tracking World States with Language Models: State-Based Evaluation Using Chess
• Using Machine Learning to Generate, Clarify, and Improve Economic Models
• Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challenges
• What Does it Mean for a Neural Network to Learn a "World Model"?

Discussions

• Evaluating the world model implicit in a generative model [hn, 159 points, 45 comments]
• Thank you Nature and @anilananth.bsky.social for this great feature on LLMs and AGI (and for highlighting our work arxiv.org/abs/2406.03689) [bsky, 10 points, 0 comments]
• I hope the boosters of physics-less AI "atmospheric models" trained on actual physics based models in climate & weather understand the limitations of what they're dealing with. I fear many do not. a [bsky, 9 points, 0 comments]
• Wo die KI planlos ist - Navigationstest und Logikspiel entlarven fehlendes Weltmodell der künstlichen Intelligenz [lemmy, 3 points, 0 comments]
• Evaluating the World Model Implicit in a Generative Model [hn, 3 points, 0 comments]
• @johnsmith4real.bsky.social : This is that paper I mentioned but subsequently couldn't find: :D arxiv.org/abs/2406.03689 [bsky, 3 points, 1 comments]
• Evaluating the World Model Implicit in a Generative Model (Harvard & MIT, November 2024) Paper: arxiv.org/abs/2406.03689 Abstract: “Recent work suggests that large language models may implicitly lear [bsky, 2 points, 0 comments]
• Smells like IPO pump and dump fodder to me. LLMs do not, and cannot, produce output with greater entropy than is in the training set. The recent Vafa result demonstrates that current architectures h [bsky, 2 points, 0 comments]
• Many assume LLMs and RLs build world models given their strength on tasks like “logical reasoning, geographic navigation, game-playing, and chemistry”. But a new paper gave the AI problems to solve wi [bsky, 1 points, 1 comments]
• Attention isn’t all you need. arxiv.org/pdf/2406.03689 [bsky, 0 points, 0 comments]
• This study investigates large language models' implicit learning of world models and proposes evaluation metrics from the Myhill-Nerode theorem. Generative models excel on standard tests but their wor [bsky, 0 points, 0 comments]
• The implicit 'world model' in an LLM is inconsistent and brittle. arxiv.org/abs/2406.03689 [bsky, 0 points, 0 comments]
• Evaluating the World Model Implicit in a Generative Model arxiv.org/abs/2406.03689 [bsky, 0 points, 0 comments]
• Evaluating the world model implicit in a generative model view on hacker news [bsky, 0 points, 0 comments]
• Evaluating the world model implicit in a generative model https://arxiv.org/abs/2406.03689 https://news.ycombinator.com/item?id=42073801 [bsky, 0 points, 0 comments]
• Evaluating the world model implicit in a generative model [bsky, 0 points, 0 comments]
• Breaking News! "Evaluating the world model implicit in a generative model" #BreakingNews #News #CurrentEvents Read more: arxiv.org/abs/2406.03689 [bsky, 0 points, 0 comments]
• arxiv.org/pdf/2406.03689 [bsky, 0 points, 0 comments]

Related

• Between world models and model worlds: on generality, agency, and worlding in machine learning
• Understanding World or Predicting Future? A Comprehensive Survey of World Models
• From Masks to Worlds: A Hitchhiker's Guide to World Models
• Toward Stable World Models: Measuring and Addressing World Instability in Generative Environments
• Making Large Language Models into World Models with Precondition and Effect Knowledge