UCD: Unlearning in LLMs via Contrastive Decoding

2025/06/12 by Vinith M. Suriyakumar, Suriyakumar, Vinith M., Ayush Sekhari +3 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematics, Computing, and Information Processing #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2506.12097

openalex publication_date 2025/06/12 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Machine unlearning aims to remove specific information, e.g. sensitive or undesirable content, from large language models (LLMs) while preserving overall performance. We propose an inference-time unlearning algorithm that uses contrastive decoding, leveraging two auxiliary smaller models, one trained without the forget set and one trained with it, to guide the outputs of the original model using their difference during inference. Our strategy substantially improves the tradeoff between unlearning effectiveness and model utility. We evaluate our approach on two unlearning benchmarks, TOFU and MUSE. Results show notable gains in both forget quality and retained performance in comparison to prior approaches, suggesting that incorporating contrastive decoding can offer an efficient, practical avenue for unlearning concepts in large-scale models.

Citations

• Exact Unlearning of Finetuning Data via Model Merging at Scale
• Open Problems in Machine Unlearning for AI Safety
• GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
• An Adversarial Perspective on Machine Unlearning for AI Safety
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuning
• Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs
• Large Language Model Unlearning via Embedding-Corrupted Prompts
• Large Scale Knowledge Washing
• SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning
• Rethinking LLM Memorization through the Lens of Adversarial Compression
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Guardrail Baselines for Unlearning in LLMs
• Rethinking Machine Unlearning for Large Language Models
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• Detecting Pretraining Data from Large Language Models
• Copyright Violations and Large Language Models
• Large Language Model Unlearning
• In-Context Unlearning: Language Models as Few Shot Unlearners
• Knowledge Sanitization of Large Language Models
• Contrastive Decoding Improves Reasoning in Large Language Models
• Ticketed Learning-Unlearning Schemes
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Model evaluation for extreme risks
• Editing Large Language Models: Problems, Methods, and Opportunities
• Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models
• Text-to-image Diffusion Models in Generative AI: A Survey
• Towards Unbounded Machine Unlearning
• Privacy Adhering Machine Un-learning in NLP
• Editing Models with Task Arithmetic
• Contrastive Decoding: Open-ended Text Generation as Optimization
• Mass-Editing Memory in a Transformer
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Algorithms that Approximate Data Removal: New Results and Limitations
• A Survey of Machine Unlearning
• Memory-Based Model Editing at Scale
• Continual Learning and Private Unlearning
• Locating and Editing Factual Associations in GPT
• Adaptive Machine Unlearning
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Extracting Training Data from Large Language Models
• Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
• Machine Unlearning
• The Curious Case of Neural Text Degeneration

Cited by

• Training-Free Adaptation of New-Generation LLMs using Legacy Clinical Models
• Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning

Related

• Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
• UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• In-Context Unlearning: Language Models as Few Shot Unlearners
• BLUR: A Bi-Level Optimization Approach for LLM Unlearning