Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

2022/10/24 by Kenneth Li, Li, Kenneth, Aspen K. Hopkins +10 · 10 voices · 52 citations
Computer Science · Social Sciences · #Computational and Text Analysis Methods #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2210.13382

arxiv published 2022/10/24 · arxiv updated 2024/06/26

Abstract

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

Cited by

• Context Is King: How In-Context Specification Shapes the Geometry of Concepts
• Rigorous Interpretation Is a Form of Evaluation
• A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
• Responsible Intelligence in Practice: A Fairness Audit of Open Large Language Models for Library Reference Services
• In-Context Algebra
• When sufficiency is insufficient: the functional information bottleneck for identifying probabilistic neural representations
• State over Tokens: Characterizing the Role of Reasoning Tokens
• Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Better World Models Can Lead to Better Post-Training Performance
• RNNs perform task computations by dynamically warping neural representations
• Emergent Riemannian geometry over learning discrete computations on continuous manifolds
• PragWorld: A Benchmark Evaluating LLMs' Local World Model under Minimal Linguistic Alterations and Conversational Dynamics
• REFLEX: Self-Refining Explainable Fact-Checking via Disentangling Truth into Style and Substance
• Beyond World Models: Rethinking Understanding in AI Models
• Next-Latent Prediction Transformers Learn Compact World Models
• Deep neural networks and humans both benefit from compositional language structure
• Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimization
• TweetyBERT: Automated parsing of birdsong through self-supervised machine learning
• Two-Layer Linear Auto-Regressive Models Estimate Latent States
• Use and usability: concepts of representation in philosophy, neuroscience, cognitive science, and computer science
• Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models
• Sequences of Logits Reveal the Low Rank Structure of Language Models
• Automatically Finding Rule-Based Neurons in OthelloGPT
• Probing Neural Combinatorial Optimization Models
• A Two-Step, Multidimensional Account of Deception in Language Models
• Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
• Post-Norm can Resharpen Attention
• Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
• Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling
• Feature Identification via the Empirical NTK
• Emergent World Representations in OpenVLA
• The problem of alignment
• Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs
• Internal Planning in Language Models: Characterizing Horizon and Branch Awareness
• Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error
• Prophecy: Inferring Formal Properties from Neuron Activations
• Generative AI for Economic Research: Use Cases and Implications for Economists
• From Prediction to Understanding: Will AI Foundation Models Transform Brain Science?
• Divergences between Language Models and Human Brains
• From Next Token Prediction to (STRIPS) World Models -- Preliminary Results
• Statistical Methods in Generative AI
• Delta Activations: A Representation for Finetuned Large Language Models
• On the possibility of deep alignment
• Tracking World States with Language Models: State-Based Evaluation Using Chess
• How Causal Abstraction Underpins Computational Explanation
• SSRL: Self-Search Reinforcement Learning
• How Does Controllability Emerge In Language Models During Pretraining?
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model
• What Does it Mean for a Neural Network to Learn a "World Model"?
• The Blessing and Curse of Dimensionality in Safety Alignment

Discussions

• Emergent World Representations in Neural Networks [hn, 3 points, 1 comments]
• A minor note, while I agree with your overall points, I think it is an open research question whether LLMs lack internal models. For example: arxiv.org/abs/2210.13382 www.anthropic.com/research/map. [bsky, 2 points, 1 comments]
• This 2022 paper uses hidden-layer-probing to show that a GPT-2 style model trained on move sequences for the game Othello internalizes a model of the game. They then screw with the internal model to p [bsky, 1 points, 1 comments]
• In my recent conversation with Dr Pietro Mazzaglia, he pointed me to this paper, which shows how a world representation emerges in GPT models without being trained on the world. Fascinating! arxiv.or [bsky, 1 points, 1 comments]
• A recent study explores a GPT model trained on predicting Othello moves, revealing an internal representation of the game state that aids in controlling outputs and generating 'latent saliency maps.' [bsky, 0 points, 0 comments]
• Ce n’est pas entièrement clair. Un LLM entraîné à compléter des suites coups décrivant des parties d’othello à développer dans ces couches internes une représentation du plateau. arxiv.org/abs/2210.13 [bsky, 0 points, 1 comments]
• ...then its internal weights start organizing into a model of the Othello board. Other recent papers showed the same. Next token prediction leads to world models. arxiv.org/abs/2210.13382 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2210.13382 a schema of a board game (x) found to emerge with only legal moves provided (y) - not sure if that was what you were after [bsky, 0 points, 0 comments]
• it can think. LLMs have an internal world model that can predict game board states: arxiv.org/abs/2210.13382 More proof: arxiv.org/pdf/2403.154... Even more proof by Max Tegmark (renowned MIT profe [bsky, 0 points, 0 comments]
• Maybe I should have elaborated further, but note that there's pretty good evidence that LMs can develop world models (arxiv.org/pdf/2210.13382), even if they are sometimes sloppy (see keyonvafa.com 's [bsky, 0 points, 0 comments]

Related

• Emergent Linear Representations in World Models of Self-Supervised Sequence Models
• Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models
• Navigating Human Language Models with Synthetic Agents
• How GPT learns layer by layer
• Revisiting the Othello World Model Hypothesis