Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering

2023/07/11 by Pengfei Li, Li, Pengfei, Gang Liu +7 · 7 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Feature (linguistics) #Generalization #Image (mathematics) #Language model #Linguistics #Machine learning #Matching (statistics) #Multimodal Machine Learning Applications #Natural language processing #Pattern recognition (psychology) #Question answering #Task (project management)

paper · pdf · doi:10.48550/arxiv.2307.05314

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/07/11 · openalex created_date 2023/07/13 · openalex updated_date 2026/07/28

Abstract

Medical visual question answering (VQA) is a challenging task that requires answering clinical questions of a given medical image, by taking consider of both visual and language information. However, due to the small scale of training data for medical VQA, pre-training fine-tuning paradigms have been a commonly used solution to improve model generalization performance. In this paper, we present a novel self-supervised approach that learns unimodal and multimodal feature representations of input images and text using medical image caption datasets, by leveraging both unimodal and multimodal contrastive losses, along with masked language modeling and image text matching as pretraining objectives. The pre-trained model is then transferred to downstream medical VQA tasks. The proposed approach achieves state-of-the-art (SOTA) performance on three publicly available medical VQA datasets with significant accuracy improvements of 2.2%, 14.7%, and 1.7% respectively. Besides, we conduct a comprehensive analysis to validate the effectiveness of different components of the approach and study different pre-training settings. Our codes and models are available at https://github.com/pengfeiliHEU/MUMC.

Cited by

• CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering
• Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering
• A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models
• Computed Tomography Visual Question Answering with Cross-modal Feature Graphing
• See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis
• Bridging the Semantic Gaps: Improving Medical VQA Consistency with LLM-Augmented Question Sets
• A Lightweight Large Vision-language Model for Multimodal Medical Images

Related

• Self-supervised vision-language pretraining for Medical visual question answering
• MISS: A Generative Pretraining and Finetuning Approach for Med-VQA
• Visual Question Answering in the Medical Domain
• Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
• MMBERT: Multimodal BERT Pretraining for Improved Medical VQA