Approximate Domain Unlearning for Vision-Language Models

2025/10/09 by Kawamura, Kodai, Y Goto, Goto, Yuta +6
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #I.2.6 #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2510.08132

openalex publication_date 2025/10/09 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information beyond the requirements of specific downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on class unlearning, where a VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driving system should accurately recognize real cars while avoiding misrecognition of illustrated cars depicted in roadside advertisements as real cars, which could be hazardous. In this paper, we introduce Approximate Domain Unlearning (ADU), a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., illustration) while preserving accuracy for other domains (e.g., real). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain distributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specific domain information. Extensive experiments show that our approach outperforms baselines built upon VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code: https://kodaikawamura.github.io/Domain_Unlearning/.

Citations

• Black-Box Forgetting
• Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement
• Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
• Transitive Vision-Language Prompt Learning for Domain Generalization
• LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP
• Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization
• SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation
• PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization
• Model Sparsity Can Simplify Machine Unlearning
• Sigmoid Loss for Language Image Pre-Training
• Boundary Unlearning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• MaPLe: Multi-modal Prompt Learning
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
• Learning with Recoverable Forgetting
• Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning
• Unrolling SGD: Understanding Factors Influencing Machine Unlearning
• Learning to Prompt for Vision-Language Models
• Machine Unlearning of Features and Labels
• On the Opportunities and Risks of Foundation Models
• Generalized Domain Adaptation
• Remember What You Want to Forget: Algorithms for Machine Unlearning
• Learning Transferable Visual Models From Natural Language Supervision
• Machine Unlearning via Algorithmic Stability
• Mixed-Privacy Forgetting in Deep Networks
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Amnesiac Machine Learning
• Machine Unlearning for Random Forests
• Descent-to-Delete: Gradient-Based Methods for Machine Unlearning
• Rethink Maximum Mean Discrepancy for Domain Adaptation
• Approximate Data Deletion from Machine Learning Models
• Machine Unlearning
• Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep\n Networks
• Certified Data Removal from Machine Learning Models
• Making AI Forget You: Data Deletion in Machine Learning
• Moment Matching for Multi-Source Domain Adaptation
• Learning to Generalize: Meta-Learning for Domain Generalization
• Deep Hashing Network for Unsupervised Domain Adaptation
• Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptation
• Membership Inference Attacks against Machine Learning Models
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Deep Transfer Learning with Joint Adaptation Networks
• Visual Domain Adaptation: A survey of recent advances
• Learning Transferable Features with Deep Adaptation Networks

Related

• Latent Domain Prompt Learning for Vision-Language Models
• SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders
• Generalizing Vision-Language Models with Dedicated Prompt Guidance
• Using Language to Extend to Unseen Domains
• Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification