R-TOFU: Unlearning in Large Reasoning Models

2025/05/21 by Sangyeon Yoon, Yoon, Sangyeon, Wonje Jeung +3 · 2 citations
Computer Science · #AI-based Problem Solving and Planning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Rough Sets and Fuzzy Logic

paper · pdf · doi:10.48550/arxiv.2505.15214

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Reasoning Models (LRMs) embed private or copyrighted information not only in their final answers but also throughout multi-step chain-of-thought (CoT) traces, making reliable unlearning far more demanding than in standard LLMs. We introduce Reasoning-TOFU (R-TOFU), the first benchmark tailored to this setting. R-TOFU augments existing unlearning tasks with realistic CoT annotations and provides step-wise metrics that expose residual knowledge invisible to answer-level checks. Using R-TOFU, we carry out a comprehensive comparison of gradient-based and preference-optimization baselines and show that conventional answer-only objectives leave substantial forget traces in reasoning. We further propose Reasoned IDK, a preference-optimization variant that preserves coherent yet inconclusive reasoning, achieving a stronger balance between forgetting efficacy and model utility than earlier refusal styles. Finally, we identify a failure mode: decoding variants such as ZeroThink and LessThink can still reveal forgotten content despite seemingly successful unlearning, emphasizing the need to evaluate models under diverse decoding settings. Together, the benchmark, analysis, and new baseline establish a systematic foundation for studying and improving unlearning in LRMs while preserving their reasoning capabilities.

Citations

• SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment
• SEPS: A Separability Measure for Robust Unlearning in LLMs
• Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
• SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
• Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Learning to Refuse: Towards Mitigating Privacy Risks in LLMs
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• Teach LLMs to Phish: Stealing Private Information from Language Models
• BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
• TOFU: A Task of Fictitious Unlearning for LLMs
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• In-Context Unlearning: Language Models as Few Shot Unlearners
• At Which Training Stage Does Code Data Help LLMs Reasoning?
• Qwen Technical Report
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• tasksource: A Dataset Harmonization Framework for Streamlined NLP Multi-Task Learning and Evaluation
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Extracting Training Data from Large Language Models
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Generating Informative and Diverse Conversational Responses via\n Adversarial Information Maximization
• Yara Parser: A Fast and Accurate Dependency Parser

Cited by

• R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge
• SoK: Machine Unlearning for Large Language Models

Related

• Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills
• Towards Reasoning-Preserving Unlearning in Multimodal Large Language Models
• Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning