WaterDrum: Watermarking for Data-centric Unlearning Metric

2025/05/08 by Xinyang Lu, Lu, Xinyang, Xinyuan Niu +16 · 4 citations
Computer Science · #Advanced Graph Neural Networks #Adversarial Robustness in Machine Learning #Benchmark (surveying) #Code (set theory) #Digital watermarking #Exploit #Generative Adversarial Networks and Image Synthesis #Metric (unit) #Retraining #Set (abstract data type)

paper · pdf · doi:10.48550/arxiv.2505.05064

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Large language model (LLM) unlearning is critical in real-world applications where it is necessary to efficiently remove the influence of private, copyrighted, or harmful data from some users. Existing utility-centric unlearning metrics (based on model utility) may fail to accurately evaluate the extent of unlearning in realistic settings such as when the forget and retain sets have semantically similar content and/or retraining the model from scratch on the retain set is impractical. This paper presents the first data-centric unlearning metric for LLMs called WaterDrum that exploits robust text watermarking to overcome these limitations. We introduce new benchmark datasets (with different levels of data similarity) for LLM unlearning that can be used to rigorously evaluate unlearning algorithms via WaterDrum. Our code is available at https://github.com/lululu008/WaterDrum and our new benchmark datasets are released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.

Citations

• Evaluating Deep Unlearning in Large Language Models
• Position: LLM Unlearning Benchmarks are Weak Measures of Progress
• Scalable watermarking for identifying large language model outputs
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• PostMark: A Robust Blackbox Watermark for Large Language Models
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Eight Methods to Evaluate Robust Unlearning in LLMs
• Machine Unlearning of Pre-trained Large Language Models
• Rethinking Machine Unlearning for Large Language Models
• Do Membership Inference Attacks Work on Large Language Models?
• TOFU: A Task of Fictitious Unlearning for LLMs
• A Survey of Text Watermarking in the Era of Large Language Models
• Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges
• Detecting Pretraining Data from Large Language Models
• Watermarking LLMs with Weight Quantization
• Embarrassingly Simple Text Watermarks
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Textbooks Are All You Need II: phi-1.5 technical report
• Robust Distortion-free Watermarks for Language Models
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• Robust Multi-bit Natural Language Watermarking through Invariant Features
• Towards Unbounded Machine Unlearning
• A Watermark for Large Language Models
• Editing Models with Task Arithmetic
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• A Survey of Machine Unlearning
• Evaluating Machine Unlearning via Epistemic Uncertainty
• Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
• Tracing Text Provenance via Context-Aware Lexical Substitution
• Mixed-Privacy Forgetting in Deep Networks
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks
• Membership Inference Attacks against Machine Learning Models
• An introduction to ROC analysis
• Source Attribution for Large Language Model-Generated Data

Cited by

• Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning
• Uncovering Scaling Laws for Large Language Models via Inverse Problems
• SoK: Machine Unlearning for Large Language Models
• Towards Reliable Forgetting: A Survey on Machine Unlearning Verification

Related

• Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
• A Closer Look at Machine Unlearning for Large Language Models
• WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models
• BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap
• GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models