OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models

2025/05/07 by Xu, Xiaoyu, Du, Minxin, Ye, Qingqing +1 · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.04416

Abstract

Large language models (LLMs) trained over extensive corpora risk memorizing sensitive, copyrighted, or toxic content. To address this, we propose OBLIVIATE, a robust unlearning framework that removes targeted data while preserving model utility. The framework follows a structured process: extracting target tokens, building retain sets, and fine-tuning with a tailored loss function comprising three components -- masking, distillation, and world fact. Using low-rank adapters (LoRA) ensures efficiency without compromising unlearning quality. We conduct experiments on multiple datasets, including Harry Potter series, WMDP, and TOFU, using a comprehensive suite of metrics: forget quality (via a new document-level memorization score), model utility, and fluency. Results demonstrate its effectiveness in resisting membership inference attacks, minimizing the impact on retained data, and maintaining robustness across diverse scenarios.

Citations

• FlowMalTrans: Unsupervised Binary Code Translation for Malware Detection Using Flow-Adapter Architecture
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
• FUNU: Boosting Machine Unlearning Efficiency by Filtering Unnecessary Unlearning
• Membership Inference Attacks and Defenses in Federated Learning: A Survey
• Membership Inference Attacks and Defenses in Federated Learning: A Survey
• On Memorization of Large Language Models in Logical Reasoning
• Position: LLM Unlearning Benchmarks are Weak Measures of Progress
• Erasing Conceptual Knowledge from Language Models
• MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts
• Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective
• Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
• Large Language Model Unlearning via Embedding-Corrupted Prompts
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• TOFU: A Task of Fictitious Unlearning for LLMs
• Large Language Models Relearn Removed Concepts
• Scalable Extraction of Training Data from (Production) Language Models
• Zephyr: Direct Distillation of LM Alignment
• In-Context Unlearning: Language Models as Few Shot Unlearners
• Who's Harry Potter? Approximate Unlearning in LLMs
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Locating and Editing Factual Associations in GPT
• Towards Continual Knowledge Learning of Language Models
• Adaptive Machine Unlearning
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Recent Trends in Named Entity Recognition (NER)
• Extracting Training Data from Large Language Models
• Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
• The Lipschitz Constant of Self-Attention
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Language Models as Knowledge Bases?
• Making AI Forget You: Data Deletion in Machine Learning
• Membership Inference Attacks against Machine Learning Models
• Pointer Sentinel Mixture Models
• Catastrophic Failure of LLM Unlearning via Quantization

Cited by

• Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs

Related

• A Closer Look at Machine Unlearning for Large Language Models
• GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
• RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
• Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Models
• Large Language Model Unlearning