2023/06/13 by Aakash Mishra, Rajat Mittal, Mishra, Aakash +7 · 1 citation
Medicine · #COVID-19 diagnosis using AI #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Physical sciences #Image and Video Processing (eess.IV) #Lung Cancer Diagnosis and Treatment #Machine Learning (cs.LG) #Medical Physics (physics.med-ph) #Radiomics and Machine Learning in Medical Imaging #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2306.08000
openalex publication_date 2023/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent advances in zero-shot learning have enabled the use of paired image-text data to replace structured labels, replacing the need for expert annotated datasets. Models such as CLIP-based CheXzero utilize these advancements in the domain of chest X-ray interpretation. We hypothesize that domain pre-trained models such as CXR-BERT, BlueBERT, and ClinicalBERT offer the potential to improve the performance of CLIP-like models with specific domain knowledge by replacing BERT weights at the cost of breaking the original model's alignment. We evaluate the performance of zero-shot classification models with domain-specific pre-training for detecting low-prevalence pathologies. Even though replacing the weights of the original CLIP-BERT degrades model performance on commonly found pathologies, we show that pre-trained text towers perform exceptionally better on low-prevalence diseases. This motivates future ensemble models with a combination of differently trained language models for maximal performance.