Language Models Represent Space and Time

2023/10/03 by Wes Gurnee, Max Tegmark, Gurnee, Wes +1 · 12 voices · 46 citations
Computer Science · Social Sciences · #Language and cultural evolution #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2310.02207

openalex publication_date 2023/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

Cited by

• Context Is King: How In-Context Specification Shapes the Geometry of Concepts
• Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective
• Hyperdimensional Probe: Decoding LLM Representations via Vector Symbolic Architectures
• Remapping and navigation of an embedding space via error minimization: a fundamental organizational principle of cognition in natural and artificial systems
• The Dead Salmons of AI Interpretability
• Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
• General agents contain world models
• Do Language Models Know Who Did What to Whom?
• How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?
• Stop treating `AGI' as the north-star goal of AI research
• The Lattice Representation Hypothesis of Large Language Models
• A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
• Towards Understanding Steering Strength
• Hierarchical Geometry of Cognitive States in Transformer Embedding Spaces
• Generative Actor Critic
• CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?
• ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking
• In-Context Representation Hijacking
• Better World Models Can Lead to Better Post-Training Performance
• Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
• Beyond Tokens in Language Models: Interpreting Activations through Text Genre Chunks
• Rep2Text: Decoding Full Text from a Single LLM Token Representation
• Next-Latent Prediction Transformers Learn Compact World Models
• Probing Neural Combinatorial Optimization Models
• Medical Interpretability and Knowledge Maps of Large Language Models
• The Geometry of Reasoning: Flowing Logics in Representation Space
• Understanding the Geospatial Reasoning Capabilities of LLMs: A Trajectory Recovery Perspective
• Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
• Mechanistic Interpretability of Socio-Political Frames in Language Models
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models
• Internal Planning in Language Models: Characterizing Horizon and Branch Awareness
• Bilinear relational structure fixes reversal curse and enables consistent model editing
• How Can We Characterize Human Generalization and Distinguish It From Generalization in Machines?
• RoadMind: Towards a Geospatial AI Expert for Disaster Response
• Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmark
• How close is AI to human-level intelligence?
• SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery
• Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanism
• Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation
• Can AI systems have free will?
• Disentangling concept semantics via multilingual averaging in Sparse Autoencoders
• Towards Integrated Alignment
• I Have No Mouth, and I Must Rhyme: Uncovering Internal Phonetic Representations in LLaMA 3.2
• What Does it Mean for a Neural Network to Learn a "World Model"?

Discussions

• Language Models Represent Space and Time [hn, 123 points, 186 comments]
• "Language Models Represent Space and Time." Beautiful work, speaking to the fundamental question of whether large language models also create "world models" internally. #MLsky #cssky arxiv.org/pdf/231 [bsky, 44 points, 4 comments]
• Now that major signatories of the "STOP AI FOR SIX MONTHS!" letter are cranking out "Look at what LLM's can do!" papers on arxiv, it seems the letter was more about stalling for time than averting an [bsky, 11 points, 2 comments]
• Do language models have an internal world model? A sense of time? At multiple spatiotemporal scales? In a new paper Wes Gurnee and Max Tegmark provide evidence that they do by finding a literal map o [bsky, 8 points, 1 comments]
• Some red flags here. Part of me is like “please walk over to your nearest liberal arts and sciences building.” The other part of me is like please narrow your arguments about LLMs and time. arxiv.org [bsky, 5 points, 0 comments]
• How Large Language Models learn and map space and time: arxiv.org/pdf/2310.022... They actually develop a sort of "temporal lobe", with specific clusters of neurons dedicated to spatiotemporal tasks [bsky, 5 points, 1 comments]
• Proof that LLM's learn a space-time "world model" according to this new paper. I haven't gone through it yet but it seems like it's worth a look arxiv.org/abs/2310.02207 #mlsky [bsky, 2 points, 0 comments]
• I haven’t explored this deeply but this seems a serious effort and the results are very interesting. I’d love to hear thoughts. #PhilosophyOfAI arxiv.org/abs/2310.02207 [bsky, 1 points, 0 comments]
• @alenabuyx.bsky.social In der Lanz Sendung zu KI haben sie LLMs als statistical parrots bezeichnet. Mittlerweile gibt es Studien die nahelegen, dass dem nicht so ist. Haben Sie Ihre Haltung dazu geänd [bsky, 1 points, 2 comments]
• 'Our analysis demonstrates that modern LLMs acquire structured knowledge about fundamental dimensions such as space and time' arxiv.org/abs/2310.02207 #llms #nlp #artificialintelligence [bsky, 0 points, 0 comments]
• Here's the paper arxiv.org/abs/2310.02207 [bsky, 0 points, 0 comments]
• We have multiple lines of evidence for this: the extraction of spatial world models from LLMs (arxiv.org/abs/2310.02207), the characterization of emergent world representations in toy models like Othe [bsky, 0 points, 1 comments]

Related

• Linear Spatial World Models Emerge in Large Language Models
• More than Correlation: Do Large Language Models Learn Causal Representations of Space?
• The Other Mind: How Language Models Exhibit Human Temporal Cognition
• Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
• Can LLMs Learn to Map the World from Local Descriptions?