SEPS: A Separability Measure for Robust Unlearning in LLMs

2025/05/20 by Wonje Jeung, Jeung, Wonje, Sangyeon Yoon +3 · 2 citations
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Diverse Research and Applications #FOS: Computer and information sciences #Higher Education Learning Practices

paper · pdf · doi:10.48550/arxiv.2505.14832

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine unlearning aims to selectively remove targeted knowledge from Large Language Models (LLMs), ensuring they forget specified content while retaining essential information. Existing unlearning metrics assess whether a model correctly answers retain queries and rejects forget queries, but they fail to capture real-world scenarios where forget queries rarely appear in isolation. In fact, forget and retain queries often coexist within the same prompt, making mixed-query evaluation crucial. We introduce SEPS, an evaluation framework that explicitly measures a model's ability to both forget and retain information within a single prompt. Through extensive experiments across three benchmarks, we identify two key failure modes in existing unlearning methods: (1) untargeted unlearning indiscriminately erases both forget and retain content once a forget query appears, and (2) targeted unlearning overfits to single-query scenarios, leading to catastrophic failures when handling multiple queries. To address these issues, we propose Mixed Prompt (MP) unlearning, a strategy that integrates both forget and retain queries into a unified training objective. Our approach significantly improves unlearning effectiveness, demonstrating robustness even in complex settings with up to eight mixed forget and retain queries in a single prompt.

Citations

• Towards Robust Evaluation of Unlearning in LLMs via Data Transformations
• Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
• Large Language Models Still Exhibit Bias in Long Text
• Position: LLM Unlearning Benchmarks are Weak Measures of Progress
• The Llama 3 Herd of Models
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• Large Language Model Unlearning via Embedding-Corrupted Prompts
• Instruction Tuning With Loss Over Instructions
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge
• Guardrail Baselines for Unlearning in LLMs
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Eight Methods to Evaluate Robust Unlearning in LLMs
• Learning to Edit: Aligning LLMs with Knowledge Editing
• Towards Safer Large Language Models through Machine Unlearning
• TOFU: A Task of Fictitious Unlearning for LLMs
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• In-Context Unlearning: Language Models as Few Shot Unlearners
• Who's Harry Potter? Approximate Unlearning in LLMs
• Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
• Qwen Technical Report
• Tight Bounds for Machine Unlearning via Differential Privacy
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark
• GPT-4 Technical Report
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
• Towards Adversarial Evaluations for Inexact Machine Unlearning
• Membership Inference Attacks From First Principles
• On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
• Manipulating SGD with Data Ordering Attacks
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Machine Unlearning via Algorithmic Stability
• Mixed-Privacy Forgetting in Deep Networks
• Machine Unlearning for Random Forests
• Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Certified Data Removal from Machine Learning Models
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Distilling the Knowledge in a Neural Network
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Cited by

• DUSK: Do Not Unlearn Shared Knowledge
• R-TOFU: Unlearning in Large Reasoning Models

Related

• BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap
• Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
• Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
• Soft Prompting for Unlearning in Large Language Models
• iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss