Targeted Forgetting of Image Subgroups in CLIP Models

2025/06/03 by Zeliang Zhang, Zhang, Zeliang, Gaowen Liu +7 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Medical Image Segmentation Techniques

paper · pdf · doi:10.48550/arxiv.2506.03117

openalex publication_date 2025/06/03 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Foundation models (FMs) such as CLIP have demonstrated impressive zero-shot performance across various tasks by leveraging large-scale, unsupervised pre-training. However, they often inherit harmful or unwanted knowledge from noisy internet-sourced datasets, compromising their reliability in real-world applications. Existing model unlearning methods either rely on access to pre-trained datasets or focus on coarse-grained unlearning (e.g., entire classes), leaving a critical gap for fine-grained unlearning. In this paper, we address the challenging scenario of selectively forgetting specific portions of knowledge within a class, without access to pre-trained data, while preserving the model's overall performance. We propose a novel three-stage approach that progressively unlearns targeted knowledge while mitigating over-forgetting. It consists of (1) a forgetting stage to fine-tune the CLIP on samples to be forgotten, (2) a reminding stage to restore performance on retained samples, and (3) a restoring stage to recover zero-shot capabilities using model souping. Additionally, we introduce knowledge distillation to handle the distribution disparity between forgetting, retaining samples, and unseen pre-trained data. Extensive experiments on CIFAR-10, ImageNet-1K, and style datasets demonstrate that our approach effectively unlearns specific subgroups while maintaining strong zero-shot performance on semantically similar subgroups and other categories, significantly outperforming baseline unlearning methods, which lose effectiveness under the CLIP unlearning setting.

Citations

• Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
• Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement
• Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIP
• Towards Certified Unlearning for Deep Neural Networks
• Learning to Unlearn for Robust Machine Unlearning
• Discover and Mitigate Multiple Biased Subgroups in Image Classifiers
• The Boy Who Survived: Removing Harry Potter from an LLM is harder than reported
• On Catastrophic Inheritance of Large Foundation Models
• Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer
• Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Hand
• Unlearning with Fisher Masking
• Who's Harry Potter? Approximate Unlearning in LLMs
• Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective
• One-Shot Machine Unlearning with Mnemonic Code
• Model Sparsity Can Simplify Machine Unlearning
• CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not
• A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others
• LAION-5B: An open large-scale dataset for training next generation image-text models
• CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention
• Evaluating Machine Unlearning via Epistemic Uncertainty
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Improved Fine-Tuning by Better Leveraging Pre-Training Data
• Unrolling SGD: Understanding Factors Influencing Machine Unlearning
• Machine Unlearning of Features and Labels
• LoRA: Low-Rank Adaptation of Large Language Models
• Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation
• What to Pre-Train on? Efficient Intermediate Task Selection
• Bayesian Variational Federated Learning and Unlearning in Decentralized Networks
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Learning Transferable Visual Models From Natural Language Supervision
• Machine Unlearning via Algorithmic Stability
• Variational Bayesian Unlearning
• BREEDS: Benchmarks for Subpopulation Shift
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• A Tutorial on Fisher Information

Cited by

• SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge

Related

• Selective, Controlled and Domain-Agnostic Unlearning in Pretrained CLIP: A Training- and Data-Free Approach
• Train Once, Forget Precisely: Anchored Optimization for Efficient Post-Hoc Unlearning
• Zero-Shot Class Unlearning in CLIP with Synthetic Samples
• Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning
• Erasing CLIP Memories: Non-Destructive, Data-Free Zero-Shot class Unlearning in CLIP Models