LLM Unlearning with LLM Beliefs

2025/10/22 by Kemou Li, Qizhou Wang, Li, Kemou +11 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.19422

openalex publication_date 2025/10/22 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Large language models trained on vast corpora inherently risk memorizing sensitive or harmful content, which may later resurface in their outputs. Prevailing unlearning methods generally rely on gradient ascent and its variants to lower the probability of specific target responses. However, we find that this strategy induces a critical side effect: probability mass is redistributed into high-likelihood regions, often corresponding to semantically related rephrasings of the targets. We refer to this as the squeezing effect, which explains why many methods yield merely spurious unlearning, a problem further obscured by automated metrics (e.g., ROUGE, truth ratio) that misreport actual success. To address this, we propose a bootstrapping (BS) framework that explicitly links the squeezing effect with the model's own high-confidence generations, namely its model beliefs. Since model beliefs inherently capture the very high-likelihood regions where probability mass is squeezed, incorporating them into the unlearning objective directly counters the squeezing effect. By jointly suppressing both target responses and model beliefs, BS-T (token) attenuates high-probability tokens, whereas BS-S (sequence) removes entire high-confidence generations, together achieving more thorough forgetting while preserving utility. Extensive experiments across diverse benchmarks with various model families confirm the effectiveness of our approach.

Citations

• What Is Preference Optimization Doing, How and Why?
• Towards Understanding Valuable Preference Data for Large Language Model Alignment
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
• From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
• Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy
• Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
• MTL-UE: Learning to Learn Nothing for Multi-Task Learning
• Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models
• GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs
• Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond
• A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models
• LLM Unlearning via Neural Activation Redirection
• Large Language Model Safety: A Holistic Survey
• Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?
• LLM Unlearning via Loss Adjustment with Only Forget Data
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models
• The Llama 3 Herd of Models
• Learning Dynamics of LLM Finetuning
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• Soft Prompting for Unlearning in Large Language Models
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
• Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
• Large Language Model Unlearning via Embedding-Corrupted Prompts
• Decoupling the Class Label and the Target Concept in Machine Unlearning
• SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Eight Methods to Evaluate Robust Unlearning in LLMs
• UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
• Rethinking Machine Unlearning for Large Language Models
• Machine Unlearning for Image-to-Image Generative Models
• TOFU: A Task of Fictitious Unlearning for LLMs
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• Large Language Model Unlearning
• In-Context Unlearning: Language Models as Few Shot Unlearners
• Who's Harry Potter? Approximate Unlearning in LLMs
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Can Large Language Models Be an Alternative to Human Evaluations?
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models
• Algorithms that Approximate Data Removal: New Results and Limitations
• Harnessing Out-Of-Distribution Examples via Augmenting Content and Style
• Recommendation Unlearning
• On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Machine Unlearning for Random Forests
• Measuring Massive Multitask Language Understanding
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Certified Data Removal from Machine Learning Models
• Making AI Forget You: Data Deletion in Machine Learning
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Understanding Black-box Predictions via Influence Functions
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models

Cited by

• What Is Preference Optimization Doing, How and Why?
• Towards Understanding Valuable Preference Data for Large Language Model Alignment
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models

Related

• UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets
• Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
• LLM Unlearning using Gradient Ratio-Based Influence Estimation and Noise Injection
• MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts
• Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy