Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision-Language Models

2025/11/29 by Thuraya Alzubaidi, Alzubaidi, Thuraya, Farhad R. Nezami +3
Medicine · Computer Science · #COVID-19 diagnosis using AI #Domain Adaptation and Few-Shot Learning #Artificial Intelligence in Healthcare and Education

paper · pdf · doi:10.48550/arxiv.2512.00597

Abstract

Foundation models trained via vision-language pretraining have demonstrated strong zero-shot capabilities across diverse image domains, yet their application to volumetric medical imaging remains limited. We introduce MedCT-VLM: Medical CT Vision-Language Model, a parameter-efficient vision-language framework designed to adapt large-scale CT foundation models for downstream clinical tasks. MedCT-VLM uses a parameter-efficient approach to adapt CT-CLIP, a contrastive vision-language model trained on 25,692 chest CT volumes, for multi-label pathology classification using Low-Rank Adaptation (LoRA). Rather than fine-tuning the model's 440 M parameters directly, we insert low-rank decomposition matrices into attention layers of both vision and text encoders, training only 1.67M parameters (0.38% of total). We evaluate on zero-shot classification across 18 thoracic pathologies, where the model must align CT embeddings with unseen text prompts at inference without task-specific training. LoRA fine-tuning improves mean AUROC from 61.3% to 68.9% (+7.6 pp), accuracy from 67.2% to 73.6% (+6.4 pp), and macro-F1 from 32.1% to 36.9% (+4.8 pp). These results demonstrate that parameter-efficient methods can effectively transfer large-scale pretraining to downstream medical imaging tasks, particularly for zero-shot scenarios where labeled data is scarce.

Citations

• Low-Rank Few-Shot Adaptation of Vision-Language Models
• SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
• Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
• R2GenGPT: Radiology Report Generation with Frozen LLMs
• Replace and Report: NLP Assisted Radiology Report Generation
• Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
• MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
• VL-Adapter: Parameter-Efficient Transfer Learning for\n Vision-and-Language Tasks
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• Contrastive Learning of Medical Visual Representations from Paired Images and Text

Related

• UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for\n Diverse Medical Imaging Modalities
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
• Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
• On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime