From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought

2023/06/22 by Lionel Wong, Gabriel Grand, Wong, Lionel +11 · 2 voices · 17 citations
#cs.CL #cs.AI #cs.SC

paper · pdf · doi:10.48550/arxiv.2306.12672

Abstract

How does language inform our downstream thinking? In particular, how do humans make meaning from language--and how can we leverage a theory of linguistic meaning to build machines that think in more human-like ways? In this paper, we propose rational meaning construction, a computational framework for language-informed thinking that combines neural language models with probabilistic models for rational inference. We frame linguistic meaning as a context-sensitive mapping from natural language into a probabilistic language of thought (PLoT)--a general-purpose symbolic substrate for generative world modeling. Our architecture integrates two computational tools that have not previously come together: we model thinking with probabilistic programs, an expressive representation for commonsense reasoning; and we model meaning construction with large language models (LLMs), which support broad-coverage translation from natural language utterances to code expressions in a probabilistic programming language. We illustrate our framework through examples covering four core domains from cognitive science: probabilistic reasoning, logical and relational reasoning, visual and physical reasoning, and social reasoning. In each, we show that LLMs can generate context-sensitive translations that capture pragmatically-appropriate linguistic meanings, while Bayesian inference with the generated programs supports coherent and robust commonsense reasoning. We extend our framework to integrate cognitively-motivated symbolic modules (physics simulators, graphics engines, and planning algorithms) to provide a unified commonsense thinking interface from language. Finally, we explore how language can drive the construction of world models themselves. We hope this work will provide a roadmap towards cognitive models and AI systems that synthesize the insights of both modern and classical computational perspectives.

Cited by

• What does it mean to understand language?
• Modeling Others' Minds as Code
• World Modeling with Probabilistic Structure Integration
• Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
• Analogy making as amortised model construction
• Human-Curated Data Authoring with LLMs: A Small-Data Approach to Domain Adaptation
• Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions
• Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models
• Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People
• The Philosophy of Language Models
• A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing
• TENG-Based Self-Powered Silent Speech Recognition Interface: from Assistive Communication to Immersive AR/VR Interaction
• On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
• Social World Models
• Generation and Evaluation in the Human Invention Process through the Lens of Game Design
• What Does it Mean for a Neural Network to Learn a "World Model"?
• Understanding Human Limits in Pattern Recognition: A Computational Model of Sequential Reasoning in Rock, Paper, Scissors

Discussions

• From word models to world models [hn, 100 points, 113 comments]
• Translating from Natural Language to the Probabilistic Language of Thought arxiv.org/pdf/2306.12672 [bsky, 6 points, 0 comments]

Related

• Towards General Natural Language Understanding with Probabilistic Worldbuilding
• Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models
• What Can Language Models Tell Us About Human Cognition?
• LLMs model how humans induce logically structured rules
• Mind's Eye: Grounded Language Model Reasoning through Simulation