SineProject: Machine Unlearning for Stable Vision Language Alignment

2025/11/23 by Garg, Arpit, Saratchandran, Hemanth, Lucey, Simon
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2511.18444

openalex publication_date 2025/11/23 · openalex created_date 2025/11/27 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) increasingly need to forget specific knowledge such as unsafe or private information without requiring full retraining. However, existing unlearning methods often disrupt vision language alignment, causing models to reject both harmful and benign queries. We trace this failure to the projector network during unlearning, its Jacobian becomes severely illconditioned, leading to unstable optimization and drift in cross modal embeddings. We introduce SineProject, a simple method that augments the frozen projector with sinusoidally modulated trainable parameters, improving the Jacobian's spectral conditioning and stabilizing alignment throughout unlearning. Across standard safety and privacy unlearning benchmarks using LLaVA v1.5 7B and 13B, SineProject reduces benign query refusals while achieving complete forgetting of targeted information, yielding state of the art forget retain trade offs with negligible computational overhead.

Citations

• Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product
• EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
• Leaner Transformers: More Heads, Less Depth
• Structured Initialization for Vision Transformers
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
• Always Skip Attention
• A Survey on Efficient Vision-Language Models
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning
• RandLoRA: Full-rank parameter-efficient fine-tuning of large models
• Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning
• Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench
• Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
• Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Weight Conditioning for Smooth Optimization of Neural Networks
• Zero-Shot Class Unlearning in CLIP with Synthetic Samples
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models
• MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning
• Efficient multi-prompt evaluation of LLMs
• Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models
• Efficient Learning With Sine-Activated Low-rank Matrices
• TOFU: A Task of Fictitious Unlearning for LLMs
• VILA: On Pre-training for Visual Language Models
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs
• Combating Representation Learning Disparity with Geometric Harmonization
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Visual Instruction Tuning
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• On skip connections and normalisation layers in deep optimisation
• A Survey of Machine Unlearning
• Flamingo: a Visual Language Model for Few-Shot Learning
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Backdoor Defense with Machine Unlearning
• LiT: Zero-Shot Transfer with Locked-image text Tuning
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective
• Learning Transferable Visual Models From Natural Language Supervision
• Implicit Neural Representations with Periodic Activation Functions
• Machine Unlearning
• Does Learning Require Memorization? A Short Tale about a Long Tail
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs

Related

• Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
• One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models
• Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
• MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
• Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models