Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision

2025/08/12 by Ahsan Habib Akash, Akash, Ahsan Habib, Greg Murray +11
Computer Science · Medicine · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Retinal Diseases and Treatments #Retinal Imaging and Analysis

paper · pdf · doi:10.48550/arxiv.2508.09087

openalex publication_date 2025/08/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) have achieved remarkable success on multimodal tasks such as image-text retrieval and zero-shot classification, yet they can exhibit demographic biases even when explicit protected attributes are absent during training. In this work, we focus on automated glaucoma screening from retinal fundus images, a critical application given that glaucoma is a leading cause of irreversible blindness and disproportionately affects underserved populations. Building on a reweighting-based contrastive learning framework, we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image-image embeddings, (ii) computes gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss, and (iii) applies these weights in a joint, top-k weighted objective to upweight underperforming clusters. This label-free approach adaptively targets the hardest examples, thereby reducing subgroup disparities. We evaluate our method on the Harvard FairVLMed glaucoma subset, reporting Equalized Odds Distance (EOD), Equalized Subgroup AUC (ES AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups.

Citations

• A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
• FairCLIP: Harnessing Fairness in Vision-Language Learning
• An Empirical Analysis of Parameter-Efficient Methods for Debiasing Pre-Trained Language Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• Fairness Without Demographics in Repeated Loss Minimization
• Fair prediction with disparate impact: A study of bias in recidivism\n prediction instruments
• Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments
• Equality of Opportunity in Supervised Learning
• The number of people with glaucoma worldwide in 2010 and 2020

Related

• Fairness-Aware Fine-Tuning of Vision-Language Models for Medical Glaucoma Diagnosis
• Data-Centric Label Smoothing for Explainable Glaucoma Screening from Eye Fundus Images
• GlaBoost: A multimodal Structured Framework for Glaucoma Risk Stratification
• Harvard Glaucoma Detection and Progression: A Multimodal Multitask Dataset and Generalization-Reinforced Semi-Supervised Learning
• DiffuPT: Class Imbalance Mitigation for Glaucoma Detection via Diffusion Based Generation and Model Pretraining