LUME: LLM Unlearning with Multitask Evaluations

2025/02/20 by Anil Ramakrishna, Ramakrishna, Anil, Yixin Wan +15 · 8 citations
Computer Science · #Authorship Attribution and Profiling #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.15097

openalex publication_date 2025/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.

Cited by

• RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning
• Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding
• A Survey on Unlearning in Large Language Models
• iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss
• SoK: Machine Unlearning for Large Language Models
• Mr. Snuffleupagus at SemEval-2025 Task 4: Unlearning Factual Knowledge from LLMs Using Adaptive RMU
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs

Related

• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
• Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond