Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering

2025/10/09 by Yuanhao Zou, Zou, Yuanhao, Zhaozheng Yin +1 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #Key (lock) #Knowledge representation and reasoning #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Question answering #Representation (politics) #Task (project management) #Visualization #Vocabulary

paper · pdf · doi:10.48550/arxiv.2510.08791

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/09 · openalex created_date 2025/10/14 · openalex updated_date 2026/08/05

Abstract

Medical Visual Question Answering (Med-VQA) is a challenging task that requires a deep understanding of both medical images and textual questions. Although recent works leveraging Medical Vision-Language Pre-training (Med-VLP) have shown strong performance on the Med-VQA task, there is still no unified solution for modality alignment, and the issue of hard negatives remains under-explored. Additionally, commonly used knowledge fusion techniques for Med-VQA may introduce irrelevant information. In this work, we propose a framework to address these challenges through three key contributions: (1) a unified solution for heterogeneous modality alignments across multiple levels, modalities, views, and stages, leveraging methods like contrastive learning and optimal transport theory; (2) a hard negative mining method that employs soft labels for multi-modality alignments and enforces the hard negative pair discrimination; and (3) a Gated Cross-Attention Module for Med-VQA that integrates the answer vocabulary as prior knowledge and selects relevant information from it. Our framework outperforms the previous state-of-the-art on widely used Med-VQA datasets like RAD-VQA, SLAKE, PathVQA and VQA-2019.

Citations

• LaPA: Latent Prompt Assist Model For Medical Visual Question Answering
• Medical Vision Language Pretraining: A survey
• Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training
• Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering
• Q2ATransformer: Improving Medical VQA via an Answer Querying Decoder
• Local Contrastive Learning for Medical Image Recognition
• LIMITR: Leveraging Local Information for Medical Image-Text Representation
• PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
• Advancing Radiograph Representation Learning with Masked Record Modeling
• Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training
• Self-supervised vision-language pretraining for Medical visual question answering
• MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
• Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
• Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge
• Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training
• Robust Cross-Modal Representation Learning with Progressive Self-Distillation
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• Multiple Meta-model Quantifying for Medical Visual Question Answering
• MMBERT: Multimodal BERT Pretraining for Improved Medical VQA
• Learning Transferable Visual Models From Natural Language Supervision
• SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering
• Unifying Vision-and-Language Tasks via Text Generation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• MedICaT: A Dataset of Medical Images, Captions, and Textual References
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• A Simple Framework for Contrastive Learning of Visual Representations
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• Overcoming Data Limitation in Medical Visual Question Answering
• UNITER: UNiversal Image-TExt Representation Learning
• Knowledge Enhanced Contextual Word Representations
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• Representation Learning with Contrastive Predictive Coding
• Graph Attention Networks
• Distilling the Knowledge in a Neural Network
• Выпуклая оптимизация

Cited by

• CLARITY: Medical World Model for Guiding Treatment Decisions by Modeling Context-Aware Disease Trajectories in Latent Space
• Multi-Sequence Parotid Gland Lesion Segmentation via Expert Text-Guided Segment Anything Model

Related

• CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
• Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
• MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
• ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images