Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models

2025/11/22 by Zhao, Dachuan, Li, Weiyue, Shen, Zhenda +4
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · doi:10.48550/arxiv.2511.18123

openalex publication_date 2025/11/22 · openalex created_date 2025/11/27 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) have become indispensable for multimodal reasoning, yet their representations often encode and amplify demographic biases, resulting in biased associations and misaligned predictions in downstream tasks. Such behavior undermines fairness and distorts the intended alignment between vision and language. Recent post-hoc approaches attempt to mitigate bias by replacing the most attribute-correlated embedding coordinates with neutral values. However, our systematic analysis reveals three critical limitations of this coordinate-wise approach: feature entanglement, poor cross-dataset generalization, and incomplete bias removal. We find that bias is not localized to a few coordinates but is instead distributed across a few linear subspaces. To address these limitations, we propose Subspace Projection Debiasing (SPD), a geometrically principled framework that identifies and removes the entire subspace of linearly decodable bias while reinserting a neutral mean component to preserve semantic fidelity. Extensive experiments across zero-shot classification, text-to-image retrieval, and image generation validate the effectiveness of SPD: our method achieves more robust debiasing with an average improvement of 18.5% across four fairness metrics, while maintaining minimal loss in task performance compared to the best debiasing baseline.

Citations

• Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation
• LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
• Joint Vision-Language Social Bias Removal for CLIP
• BendVLM: Test-Time Debiasing of Vision-Language Embeddings
• Identifying Implicit Social Biases in Vision-Language Models
• A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
• SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP
• xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
• Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
• Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
• They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
• FairCLIP: Harnessing Fairness in Vision-Language Learning
• Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images
• SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples
• Finetuning Text-to-Image Diffusion Models for Fairness
• FACET: Fairness in Computer Vision Evaluation Benchmark
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
• Mitigating Test-Time Bias for Fair Image Retrieval
• Any-to-Any Generation via Composable Diffusion
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Visual Instruction Tuning
• Model-Agnostic Gender Debiased Image Captioning
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• DeAR: Debiasing Vision-Language Models with Additive Residuals
• MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
• Debiasing Vision-Language Models via Biased Prompts
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities
• Flamingo: a Visual Language Model for Few-Shot Learning
• A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning
• DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models
• Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search
• Understanding and Evaluating Racial Biases in Image Captioning
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Women also Snowboard: Overcoming Bias in Captioning Models
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Deep Residual Learning for Image Recognition
• Representation Learning: A Review and New Perspectives
• Representation Learning: A Review and New Perspectives
• Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflection

Related

• Debiasing Vision-Language Models via Biased Prompts
• DeAR: Debiasing Vision-Language Models with Additive Residuals
• A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning
• debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
• Joint Vision-Language Social Bias Removal for CLIP