Your thoughts tell who you are: Characterize the reasoning patterns of LRMs

2025/09/29 by Y Chen, Chen, Yida, Yuning Mao +17
Computer Science · Social Sciences · #Analytic reasoning #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Case-based reasoning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Focus (optics) #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #Qualitative reasoning #Reasoning system #Software Engineering Techniques and Practices #TRACE (psycholinguistics) #Task (project management) #Taxonomy (biology)

paper · pdf · doi:10.48550/arxiv.2509.24147

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/08/05

Abstract

Current comparisons of large reasoning models (LRMs) focus on macro-level statistics such as task accuracy or reasoning length. Whether different LRMs reason differently remains an open question. To address this gap, we introduce the LLM-proposed Open Taxonomy (LOT), a classification method that uses a generative language model to compare reasoning traces from two LRMs and articulate their distinctive features in words. LOT then models how these features predict the source LRM of a reasoning trace based on their empirical distributions across LRM outputs. Iterating this process over a dataset of reasoning traces yields a human-readable taxonomy that characterizes how models think. We apply LOT to compare the reasoning of 12 open-source LRMs on tasks in math, science, and coding. LOT identifies systematic differences in their thoughts, achieving 80-100% accuracy in distinguishing reasoning traces from LRMs that differ in scale, base model family, or objective domain. Beyond classification, LOT's natural-language taxonomy provides qualitative explanations of how LRMs think differently. Finally, in a case study, we link the reasoning differences to performance: aligning the reasoning style of smaller Qwen3 models with that of the largest Qwen3 during test time improves their accuracy on GPQA by 3.3-5.7%.

Citations

• The Few-shot Dilemma: Over-prompting Large Language Models
• Thought Anchors: Which LLM Reasoning Steps Matter?
• Seed-Coder: Let the Code Model Curate Data for Itself
• What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning
• Reinforcing General Reasoning without Verifiers
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
• Qwen3 Technical Report
• Phi-4-reasoning Technical Report
• DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• Idiosyncrasies in Large Language Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Llama 3 Herd of Models
• Verbalized Machine Learning: Revisiting Machine Learning with Language Models
• Crafting Interpretable Embeddings by Asking LLMs Questions
• Many-Shot In-Context Learning
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Automatic Prompt Optimization with "Gradient Descent" and Beam Search
• Large Language Models Are Human-Level Prompt Engineers
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Measuring Mathematical Problem Solving With the MATH Dataset
• EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers

Related

• Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinking
• THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models
• Probing the "Psyche'' of Large Reasoning Models: Understanding Through a Human Lens
• Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
• Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers