FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attention

2025/11/15 by Peng Zhang, Zhang, Peng, Zhihui Lai +7
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2511.12215

Abstract

Medical vision-language pre-training (VLP) offers significant potential for advancing medical image understanding by leveraging paired image-report data. However, existing methods are limited by False Negatives (FaNe) induced by semantically similar texts and insufficient fine-grained cross-modal alignment. To address these limitations, we propose FaNe, a semantic-enhanced VLP framework. To mitigate false negatives, we introduce a semantic-aware positive pair mining strategy based on text-text similarity with adaptive normalization. Furthermore, we design a text-conditioned sparse attention pooling module to enable fine-grained image-text alignment through localized visual representations guided by textual cues. To strengthen intra-modal discrimination, we develop a hard-negative aware contrastive loss that adaptively reweights semantically similar negatives. Extensive experiments on five downstream medical imaging benchmarks demonstrate that FaNe achieves state-of-the-art performance across image classification, object detection, and semantic segmentation, validating the effectiveness of our framework.

Citations

• MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
• FLAIR: VLM with Fine-grained Language-informed Image Representations
• CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
• MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning
• IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training
• PRIOR: Prototype Representation Joint Learning from Medical Images and Reports
• M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization
• SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger
• Sigmoid Loss for Language Image Pre-Training
• Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images
• CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and\n Expert Comparison
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• Representation Learning with Contrastive Predictive Coding
• YOLOv3: An Incremental Improvement
• Deep Residual Learning for Image Recognition
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge

Related

• SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation
• Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination
• MMCLIP: Cross-modal Attention Masked Modelling for Medical Language-Image Pre-Training
• Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis
• MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations