A Unified Framework for Diffusion Model Unlearning with f-Divergence

2025/09/25 by Nicola Novello, Novello, Nicola, Fontana, Federico +6
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #Control Systems and Identification #FOS: Computer and information sciences #Fault Detection and Control Systems #Machine Learning (cs.LG) #Matrix Theory and Algorithms

paper · pdf · doi:10.48550/arxiv.2509.21167

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most existing methods for concept unlearning in text-to-image diffusion models minimize a mean squared error (MSE) loss between the denoiser outputs conditioned on a target and an anchor concept, which is implicitly the KL divergence between two Gaussians. We generalize this objective to any f-divergence, recovering MSE as the KL instance, and identify a family of α-divergences whose Gaussian closed-form yields cheap, MSE-like training objectives. For the remaining f-divergences, we provide a min-max objective based on the variational formulation of the f-divergence. We theoretically analyze and numerically validate how different f-divergences impact the gradient magnitude and the convergence properties of the algorithm, affecting the quality of unlearning. For instance, we observe that the Hellinger closed-form instance consistently dominates MSE across multiple scenarios. More generally, the proposed unified framework offers a flexible paradigm for selecting the optimal divergence based on the application and user goal, allowing for finer control over the trade-off between unlearning efficacy and generative fidelity.

Citations

• Minimalist Concept Erasure in Generative Models
• Robust Classification with Noisy Labels Based on Posterior Maximization
• Fine-Grained Erasure in Text-to-Image Diffusion-based Foundation Models
• LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty
• Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation
• SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models
• One-step Diffusion Models with f-Divergence Distribution Matching
• Fantastic Targets for Concept Erasure in Diffusion Models and Where To Find Them
• Loss Functions and Operators Generated by f-Divergences
• Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models
• Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation
• SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation
• LLM Unlearning via Loss Adjustment with Only Forget Data
• Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting
• Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimization
• Analyzing α-divergence in Gaussian Rate-Distortion-Perception Theory
• Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient
• Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models
• Robust Semi-supervised Learning via f-Divergence and α-Rényi Divergence
• Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images
• Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond
• MACE: Mass Concept Erasure in Diffusion Models
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Rethinking Machine Unlearning for Large Language Models
• f-Divergence Based Classification: Beyond the Use of Cross-Entropy
• One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications
• Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers
• MMA-Diffusion: MultiModal Attack on Diffusion Models
• SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation
• Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?
• Unified Concept Editing in Diffusion Models
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening
• AI Art and its Impact on Artists
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• Mutual Information Estimation via f-Divergence and Data Derangements
• Understanding and Mitigating Copying in Diffusion Models
• Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models
• Ablating Concepts in Text-to-Image Diffusion Models
• Editing Implicit Assumptions in Text-to-Image Diffusion Models
• Erasing Concepts from Diffusion Models
• Towards Unbounded Machine Unlearning
• Extracting Training Data from Diffusion Models
• Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Mass-Editing Memory in a Transformer
• Flow Matching for Generative Modeling
• Red-Teaming the Stable Diffusion Safety Filter
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Learning Transferable Visual Models From Natural Language Supervision
• When Optimizing f-divergence is Robust with Label Noise
• f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning
• Denoising Diffusion Probabilistic Models
• Training Deep Energy-Based Models with f-Divergence Minimization
• Gradient Surgery for Multi-Task Learning
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Demystifying MMD GANs
• Gradient descent GAN optimization is locally stable
• f-GAN: Training Generative Neural Samplers using Variational Divergence\n Minimization
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Related

• Concept Unlearning by Modeling Key Steps of Diffusion Process
• Model Integrity when Unlearning with T2I Diffusion Models
• Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient
• UnGuide: Learning to Forget with LoRA-Guided Diffusion Models