Towards Evaluation for Real-World LLM Unlearning

2025/08/02 by Miao, Ke, Hu, Yuke, Li, Xiaochen +4
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.01324

Abstract

This paper analyzes the limitations of existing unlearning evaluation metrics in terms of practicality, exactness, and robustness in real-world LLM unlearning scenarios. To overcome these limitations, we propose a new metric called Distribution Correction-based Unlearning Evaluation (DCUE). It identifies core tokens and corrects distributional biases in their confidence scores using a validation set. The evaluation results are quantified using the Kolmogorov-Smirnov test. Experimental results demonstrate that DCUE overcomes the limitations of existing metrics, which also guides the design of more practical and reliable unlearning algorithms in the future.

Citations

• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
• Fantastic Copyrighted Beasts and How (Not) to Generate Them
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• Towards Efficient and Effective Unlearning of Large Language Models for Recommendation
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Guardrail Baselines for Unlearning in LLMs
• Eight Methods to Evaluate Robust Unlearning in LLMs
• TOFU: A Task of Fictitious Unlearning for LLMs
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models
• In-Context Unlearning: Language Models as Few Shot Unlearners
• Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• Large Language Models in Finance: A Survey
• Knowledge Sanitization of Large Language Models
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
• Large language models in medicine
• Ticketed Learning-Unlearning Schemes
• Composing Parameter-Efficient Modules with Arithmetic Operations
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Foundation Models and Fair Use
• Privacy Adhering Machine Un-learning in NLP
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Adaptive Machine Unlearning
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Extracting Training Data from Large Language Models
• Approximate Data Deletion from Machine Learning Models
• Machine Unlearning
• Making AI Forget You: Data Deletion in Machine Learning
• Catastrophic Failure of LLM Unlearning via Quantization

Related

• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
• Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
• Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosis
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning