Words That Make Language Models Perceive

2025/10/02 by S. Q. Wang, Sophie L. Wang, Phillip Isola +4 · 4 voices · 4 citations
Computer Science · #Explainable Artificial Intelligence (XAI) #Generative Adversarial Networks and Image Synthesis #Language model #Multimodal Machine Learning Applications #Perception #Sensory cue #Sensory system #Test (biology) #Visual perception

paper · pdf · doi:10.48550/arxiv.2510.02425

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Large language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by multimodal regularities encoded in language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational alignment with specialist vision and audio encoders. When a sensory prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.

Citations

• The Platonic Representation Hypothesis
• DINOv2: Learning Robust Visual Features without Supervision

Cited by

• Large Language Models Report Subjective Experience Under Self-Referential Processing
• Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
• Learning to Draw ASCII Improves Spatial Reasoning in Language Models
• Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

Discussions

• Over the past year, my lab has been working on fleshing out theory + applications of the Platonic Representation Hypothesis. Today I want to share two new works on this topic: Eliciting higher alignme [bsky, 133 points, 1 comments]
• Words That Make Language Models Perceive [hn, 3 points, 0 comments]
• arxiv.org/abs/2510.02425 [bsky, 2 points, 1 comments]
• Words that make language models perceive [hn, 2 points, 0 comments]

Related

• With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models
• Large Language Models Implicitly Learn to See and Hear Just By Reading
• Language Models Don't Learn the Physical Manifestation of Language
• Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings
• What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models