LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

2026/04/07 by Lihao Sun, Hang Dong, Bo Qiao +3 · 1 voice · 1 citation
Computer Science · #Automated reasoning #Convergence (economics) #Correctness #Linear subspace #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Non-monotonic logic #Reasoning system #Representation (politics) #Separable space #Spatial intelligence #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · open access · doi:10.48550/arxiv.2604.05655

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/04/07 · arxiv published 2026/04/07 · arxiv updated 2026/04/07 · openalex created_date 2026/04/09 · openalex updated_date 2026/07/28

Abstract

This work characterizes large language models' chain-of-thought generation as a structured trajectory through representation space. We show that mathematical reasoning traverses functionally ordered, step-specific subspaces that become increasingly separable with layer depth. This structure already exists in base models, while reasoning training primarily accelerates convergence toward termination-related subspaces rather than introducing new representational organization. While early reasoning steps follow similar trajectories, correct and incorrect solutions diverge systematically at late stages. This late-stage divergence enables mid-reasoning prediction of final-answer correctness with ROC-AUC up to 0.87. Furthermore, we introduce trajectory-based steering, an inference-time intervention framework that enables reasoning correction and length control based on derived ideal trajectories. Together, these results establish reasoning trajectories as a geometric lens for interpreting, predicting, and controlling LLM reasoning behavior.

Citations

• Valence-Arousal Subspace in LLMs: Circular Emotion Geometry and Multi-Behavioral Control
• The Geometry of Reasoning: Flowing Logics in Representation Space
• LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
• Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet
• Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models
• Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
• AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
• The Geometry of Self-Verification in a Task-Specific Reasoning Model
• Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification
• Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
• The Llama 3 Herd of Models
• Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
• Steering Language Models With Activation Engineering
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
• Training Verifiers to Solve Math Word Problems
• Measuring Mathematical Problem Solving With the MATH Dataset
• Attention Is All You Need

Cited by

• The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics

Discussions

• 📢 Accepted to #ACL2026 Main Conference! See you in San Diego. Thanks to all collaborators at Microsoft: Hang Dong, Bo Qiao, Qingwei Lin, Dongmei Zhang, and Saravan Rajmohan. Paper: arxiv.org/pdf/2604 [bsky, 1 points, 0 comments]

Related

• Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
• DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
• Reasoning LLMs are Wandering Solution Explorers
• Thought Anchors: Which LLM Reasoning Steps Matter?
• The Geometry of Reasoning: Flowing Logics in Representation Space