On the Scaling Laws of Geographical Representation in Language Models

2024/02/29 by Nathan Godey, Godey, Nathan, Eric Villemonte de La Clergerie +3 · 7 citations
Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Geographic Information Systems Studies #Human Mobility and Location-Based Analysis #Language and cultural evolution

paper · pdf · doi:10.48550/arxiv.2402.19406

openalex publication_date 2024/02/29 · openalex created_date 2024/03/06 · openalex updated_date 2026/07/28

Abstract

Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited by extending this result to Large Language Models (LLMs). In this paper, we propose to fill the gap between well-established and recent literature by observing how geographical knowledge evolves when scaling language models. We show that geographical knowledge is observable even for tiny models, and that it scales consistently as we increase the model size. Notably, we observe that larger language models cannot mitigate the geographical bias that is inherent to the training data.

Cited by

• Coordinates from Context: Using LLMs to Ground Complex Location References
• From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
• Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
• Geospatial Mechanistic Interpretability of Large Language Models
• Revealing economic facts: LLMs know more than they say
• FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation
• TraveLLaMA: A Multimodal Travel Assistant with Large-Scale Dataset and Structured Reasoning

Related

• Geospatial Mechanistic Interpretability of Large Language Models
• Evaluation of Geographical Distortions in Language Models
• Geographic and Geopolitical Biases of Language Models
• Large Language Models are Geographically Biased
• Are Large Language Models Geospatially Knowledgeable?