A Mechanistic Perspective and Difficulty Metric for Unlearning

2026/01/01 by Jiali Cheng, Ziheng Chen, Chirag Agarwal +1
Computer Science · Decision Sciences · #Machine Learning and Algorithms #Advanced Bandit Algorithms Research #Optimization and Search Problems

paper · pdf · doi:10.18653/v1/2026.findings-acl.532

Abstract

Machine unlearning is becoming essential for building trustworthy and compliant language models.Yet unlearning success varies considerably across individual samples: some are reliably erased, while others persist despite the same procedure.We argue that this disparity is not only a data-side phenomenon, but also reflects model-internal mechanisms that encode and protect memorized information.We study this problem from a mechanistic perspective based on model circuits-structured interaction pathways that govern how predictions are formed.We propose Circuit-guided Unlearning Difficulty (CUD), a pre-unlearning metric that assigns each sample a continuous difficulty score using circuit-level signals.Extensive experiments demonstrate that CUD reliably separates intrinsically easy and hard samples, and remains stable across unlearning methods.We identify key circuit-level patterns that reveal a mechanistic signature of unlearning difficulty: easy-to-unlearn samples are associated with shorter, shallower interactions concentrated in earlier-to-intermediate parts of the original model, whereas hard-to-unlearn samples rely on longer and deeper pathways closer to late-stage computation.Compared to existing qualitative studies, CUD takes a first step toward a principled, fine-grained, and interpretable analysis of unlearning difficulty; and motivates the development of unlearning methods grounded in model mechanisms.

Citations

• CLUE: Conflict-guided Localization for LLM Unlearning Framework
• FUTURE: Flexible Unlearning for Tree Ensemble
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• BLUR: A Bi-Level Optimization Approach for LLM Unlearning
• Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
• Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
• LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
• A Neuro-inspired Interpretation of Unlearning in Large Language Models through Sample-level Unlearning Difficulty
• Understanding Machine Unlearning Through the Lens of Mode Connectivity
• Understanding Machine Unlearning Through the Lens of Mode Connectivity
• Not All Data Are Unlearned Equally
• Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond
• Tool Unlearning for Tool-Augmented LLMs
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
• Instance-Level Difficulty: A Missing Perspective in Machine Unlearning
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• Intrinsic Test of Unlearning Using Parametric Knowledge Traces
• What Makes Language Models Good-enough?
• What makes unlearning hard and what to do about it
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms
• Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning
• Towards Efficient and Effective Unlearning of Large Language Models for Recommendation
• UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
• TOFU: A Task of Fictitious Unlearning for LLMs
• SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation
• Attribution Patching Outperforms Automated Circuit Discovery
• Who's Harry Potter? Approximate Unlearning in LLMs
• Towards Automated Circuit Discovery for Mechanistic Interpretability
• Locating and Editing Factual Associations in GPT
• Causal Analysis of Syntactic Agreement Mechanisms in Neural Language\n Models
• Machine Unlearning
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Related

• A Neuro-inspired Interpretation of Unlearning in Large Language Models through Sample-level Unlearning Difficulty
• Instance-Level Difficulty: A Missing Perspective in Machine Unlearning
• Layered Unlearning for Adversarial Relearning
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
• What makes unlearning hard and what to do about it