Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models

2025/05/20 by Zahraa Al Sahili, Sahili, Zahraa Al, Ioannis Patras +3 · 1 citation
Psychology · #Categorization, perception, and language #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language, Metaphor, and Cognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2505.14160

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multilingual vision-language models (VLMs) promise universal image-text retrieval, yet their social biases remain underexplored. We perform the first systematic audit of four public multilingual CLIP variants: M-CLIP, NLLB-CLIP, CAPIVARA-CLIP, and the debiased SigLIP-2, covering ten languages that differ in resource availability and morphological gender marking. Using balanced subsets of FairFace and the PATA stereotype suite in a zero-shot setting, we quantify race and gender bias and measure stereotype amplification. Contrary to the intuition that multilinguality mitigates bias, every model exhibits stronger gender skew than its English-only baseline. CAPIVARA-CLIP shows its largest biases precisely in the low-resource languages it targets, while the shared encoder of NLLB-CLIP and SigLIP-2 transfers English gender stereotypes into gender-neutral languages; loosely coupled encoders largely avoid this leakage. Although SigLIP-2 reduces agency and communion skews, it inherits -- and in caption-sparse contexts (e.g., Xhosa) amplifies -- the English anchor's crime associations. Highly gendered languages consistently magnify all bias types, yet gender-neutral languages remain vulnerable whenever cross-lingual weight sharing imports foreign stereotypes. Aggregated metrics thus mask language-specific hot spots, underscoring the need for fine-grained, language-aware bias evaluation in future multilingual VLM research.

Citations

• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework
• Identifying Implicit Social Biases in Vision-Language Models
• FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models
• Do Multilingual Large Language Models Mitigate Stereotype Bias?
• MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
• FairCLIP: Harnessing Fairness in Vision-Language Learning
• FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs
• NLLB-CLIP -- train performant multilingual image retrieval model on a budget
• Large Language Models
• Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale
• Comparing Biases and the Impact of Multilingual Training across Multiple Languages
• DeAR: Debiasing Vision-Language Models with Additive Residuals
• Debiasing Vision-Language Models via Biased Prompts
• Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset
• Learning Transferable Visual Models From Natural Language Supervision
• Evaluating Gender Bias in Machine Translation

Cited by

• Investigating Social Bias in Narrative Image Generation

Related

• Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models
• BAFIS (Battle-Arena for Fair Image Synthesis)
• Vision-Language Models display a strong gender bias
• Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
• Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities