Unsupervised Multimodal Representation Learning across Medical Images and Reports

2018/11/21 by Tzu-Ming Harry Hsu, Wei‐Hung Weng, Hsu, Tzu-Ming Harry +7 · 2 citations
Computer Science · #Multimodal Machine Learning Applications #Domain Adaptation and Few-Shot Learning #AI in cancer detection

paper · pdf · doi:10.48550/arxiv.1811.08615

Abstract

Joint embeddings between medical imaging modalities and associated radiology reports have the potential to offer significant benefits to the clinical community, ranging from cross-domain retrieval to conditional generation of reports to the broader goals of multimodal representation learning. In this work, we establish baseline joint embedding results measured via both local and global retrieval methods on the soon to be released MIMIC-CXR dataset consisting of both chest X-ray images and the associated radiology reports. We examine both supervised and unsupervised methods on this task and show that for document retrieval tasks with the learned representations, only a limited amount of supervision is needed to yield results comparable to those of fully-supervised methods.

Citations

• NLTK: The Natural Language Toolkit

Cited by

• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
• Clinically Accurate Chest X-Ray Report Generation

Related

• Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
• Leveraging LLMs for Multimodal Retrieval-Augmented Radiology Report Generation via Key Phrase Extraction
• Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentation