A Statistical Physics of Language Model Reasoning

2025/06/04 by Jack David Carson, Carson, Jack David, Amir Reisizadeh +1 · 2 voices
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language and cultural evolution #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.04374

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-level hidden state trajectories as a stochastic dynamical system on a lower-dimensional manifold. This drift-diffusion system uses latent regime switching to capture diverse reasoning phases, including misaligned states or failures. Empirical trajectories (8 models, 7 benchmarks) show a rank-40 projection (balancing variance capture and feasibility) explains ~50% variance. We find four latent reasoning regimes. An SLDS model is formulated and validated to capture these features. The framework enables low-cost reasoning simulation, offering tools to study and predict critical transitions like misaligned states or other LM failures.

Discussions

• what if statistical physics worked? arxiv.org/abs/2506.04374 [bsky, 8 points, 1 comments]
• arxiv.org/abs/2506.04374 [bsky, 0 points, 0 comments]

Related

• Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics
• Modeling Hierarchical Thinking in Large Reasoning Models
• Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics
• Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering
• CTRLS: Chain-of-Thought Reasoning via Latent State-Transition