Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps

2025/02/20 by Martin Tutek, Fateme Hashemi Chaleshtori, Tutek, Martin +5 · 2 voices · 12 citations
Social Sciences · #Education and Islamic Studies #Religion, Spirituality, and Psychology #cs.CL

paper · pdf · doi:10.48550/arxiv.2502.14829

openalex publication_date 2025/02/20 · openalex created_date 2025/02/22 · openalex updated_date 2026/07/28

Abstract

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.

Citations

• AlphaMath Almost Zero: Process Supervision without Process
• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Who's Harry Potter? Approximate Unlearning in LLMs
• Training language models to follow instructions with human feedback
• Machine Unlearning

Cited by

• Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutors
• CASE: Causal Alignment and Structural Enforcement for Improving Chain-of-Thought Faithfulness
• MUX: Continuous Reasoning via Multiplexed Tokens
• Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization
• Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Verifying Chain-of-Thought Reasoning via Its Computational Graph
• Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
• FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
• Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
• Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models
• Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?

Discussions

• 🚨🚨 New preprint 🚨🚨 Ever wonder whether verbalized CoTs correspond to the internal reasoning process of the model? We propose a novel parametric faithfulness approach, which erases information co [bsky, 48 points, 2 comments]
• As in a version of this talk a few months back, I'll cover our #EMNLP2025 unlearning-based approach for measuring CoT parametric faithfulness led by @mtutek.bsky.social. Also presented by @boknilev.bs [bsky, 3 points, 0 comments]

Related

• Measuring Faithfulness in Chain-of-Thought Reasoning
• Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
• Reasoning Models Better Express Their Confidence
• Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?
• FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness