Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

2025/04/14 by Subbarao Kambhampati, Karthik Valmeekam, Kambhampati, Subbarao +16 · 17 voices · 20 citations
Computer Science · Psychology · #Encoding (memory) #Human language #Innovative Teaching and Learning Methods #Intelligent Tutoring Systems and Adaptive Learning #Language model #Natural language generation #Security token #Text generation #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.09762

published in ArXiv.org

openalex publication_date 2025/04/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Intermediate token generation (ITG), where a model produces output before the solution, has become a standard method to improve the performance of language models on reasoning tasks. These intermediate tokens have been called \sayreasoning traces or even \saythinking traces -- implicitly anthropomorphizing the traces, and implying that these traces resemble steps a human might take when solving a challenging problem, and as such can provide an interpretable window into the operation of the model's thinking process to the end user. In this position paper, we present evidence that this anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research. We call on the community to avoid such anthropomorphization of intermediate tokens.

Citations

• Reasoning Models Generate Societies of Thought
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
• s1: Simple test-time scaling
• The Lessons of Developing Process Reward Models in Mathematical Reasoning
• Reasoning Beyond Words ? Exploring framework for hidden state reasoning
• Best-of-N Jailbreaking
• Mastering Board Games by External and Internal Planning with Language Models
• Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces
• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
• Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping
• Mathematical discoveries from program search with large language models
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
• Sparks of Artificial General Intelligence: Early experiments with GPT-4

Cited by

• Solver-Hard Is Not Model-Hard: A Hardness-Controlled Diagnostic for LLM Constraint Reasoning
• Humans Perceive Wrong Narratives from AI Reasoning Texts
• Some hypotheses on how chatbots work in problem-solving-driven conversations. Large Language Models as confirmation of the Innovation Illusion
• Masked Distillation: Internalizing the Chain-of-Thought in Language Models
• State over Tokens: Characterizing the Role of Reasoning Tokens
• On the Notion that Language Models Reason
• Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interaction
• Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning
• Trace Length is a Simple Uncertainty Signal in Reasoning Models
• Tracing Uncertainty in Language Model "Reasoning"
• Code Like Humans: A Multi-Agent Solution for Medical Coding
• The Need for Verification in AI-Driven Scientific Discovery
• Lexical Hints of Accuracy in LLM Reasoning Chains
• Retrieval-augmented reasoning with lean language models
• Feedback-Induced Performance Decline in LLM-Based Decision-Making
• Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding
• Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation
• Humans Disengage, Reasoning Models Persist: Separating Difficulty Registration from Deliberation Allocation
• Do explanations generalize across large reasoning models?
• Humans and LLMs Diverge on Probabilistic Inferences

Discussions

• We updated our position on anthropomorphization of intermediate tokens in LRMs--with additional results and a call to action.. arxiv.org/abs/2504.09762 [bsky, 19 points, 0 comments]
• "We collated emerging evidence to support our position that intermediate tokens are not guaranteed to have any end user semantics, and that their interpretability and solution accuracy are often at lo [bsky, 19 points, 2 comments]
• Boy do I have a position paper for you arxiv.org/abs/2504.09762 [bsky, 9 points, 2 comments]
• Don't use summarizers for the papers by @rao2z.bsky.social because the reasoning traces therein are, unlike the LRMs & LLMs under investigation, substantively meaningful, semantically well-ordered, an [bsky, 8 points, 1 comments]
• Subbarao Kambhampati is my favourite Ai researcher to read for a dose of realism on Ai hype. Eg Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! arxiv.org/abs/2504.09762 [bsky, 5 points, 0 comments]
• Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Trac [hn, 4 points, 0 comments]
• arxiv.org/abs/2504.09762 [bsky, 3 points, 0 comments]
• (How) Do reasoning models reason? [hn, 3 points, 0 comments]
• (How) Do reasoning models reason? [hn, 2 points, 0 comments]
• I really like this paper, even if I disagree with some aspects of it. I rant all the time about the atrocious terminological chaos of AI to deaf ears, and this paper shows one example of how damaging [bsky, 1 points, 1 comments]
• Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces [hn, 1 points, 0 comments]
• Anyway, I'll stop now. If you want to read @rao2z.bsky.social's full paper -- I will warn you it does get technical in parts -- you can find it here: arxiv.org/pdf/2504.09762 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2504.09762 here's a paper on intermediate tokens if you haven't seen it my thoughts on People Who Know Technical Stuff About AI is that they owe it to the public not to use the marketing [bsky, 0 points, 1 comments]
• https://bsky.app/profile/jake-metzger.bsky.social/post/3lqv46uwovu2y [bsky, 0 points, 0 comments]
• arxiv.org/pdf/2504.09762 I wasn't sure if that part is a scam but it seems it is [bsky, 0 points, 1 comments]
• → Do #machines "think"? → How will we cooperate with #cognitive #agents in the future? This article is an important piece in my quest to answer them. It offers detailed insight into the differences be [bsky, 0 points, 0 comments]
• A paper to be read before any more of the “the machine tried to preserve itself”. It did not. arxiv.org/abs/2504.097... [bsky, 0 points, 0 comments]

Related

• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
• Response: Emergent analogical reasoning in large language models
• Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity
• State over Tokens: Characterizing the Role of Reasoning Tokens
• Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models