Vision-language model-based semantic-guided imaging biomarker for lung nodule malignancy prediction

2025/04/30 by Luoting Zhuang, Zhuang, Luoting, Seyed Mohammad Hossein Tabatabaei +11 · 1 voice
Medicine · #Lung Cancer Diagnosis and Treatment #COVID-19 diagnosis using AI #Radiomics and Machine Learning in Medical Imaging

paper · doi:10.1016/j.jbi.2025.104947

openalex created_date 2025/10/10 · openalex publication_date 2025/10/27 · openalex updated_date 2026/07/31

Abstract

OBJECTIVE: Machine learning models have utilized semantic features, deep features, or both to assess lung nodule malignancy. However, their reliance on manual annotation during inference, limited interpretability, and sensitivity to imaging variations hinder their application in real-world clinical settings. Thus, this research aims to integrate semantic features derived from radiologists' assessments of nodules, guiding the model to learn clinically relevant, robust, and explainable imaging features for predicting lung cancer. METHODS: We obtained 938 low-dose CT scans from the National Lung Screening Trial (NLST) with 1,261 nodules and semantic features. Additionally, the Lung Image Database Consortium dataset contains 1,018 CT scans, with 2,625 lesions annotated for nodule characteristics. Three external datasets were obtained from UCLA Health, the LUNGx Challenge, and the Duke Lung Cancer Screening. For imaging input, we obtained 2D nodule slices in nine directions from 50×50×50mm nodule crop. We converted structured semantic features into sentences using Gemini. We fine-tuned a pretrained Contrastive Language-Image Pretraining (CLIP) model with a parameter-efficient fine-tuning approach to align imaging and semantic text features and predict the one-year lung cancer diagnosis. RESULTS: Our model outperformed the state-of-the-art (SOTA) models in the NLST test set with an AUROC of 0.901 and AUPRC of 0.776. It also showed robust results in external datasets. Using CLIP, we also obtained predictions on semantic features through zero-shot inference, such as nodule margin (AUROC: 0.807), nodule consistency (0.812), and pleural attachment (0.840). CONCLUSION: By incorporating semantic features into the vision-language model, our approach surpasses the SOTA models in predicting lung cancer from CT scans collected from diverse clinical settings. It provides explainable outputs, aiding clinicians in comprehending the underlying meaning of model predictions. The code is available at https://github.com/luotingzhuang/CLIP_nodule.

Citations

• The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans
• An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification
• Reduced Lung-Cancer Mortality with Volume CT Screening in a Randomized Trial
• TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning
• The Duke Lung Cancer Screening (DLCS) Dataset: A Reference Dataset of Annotated Low-Dose Screening Thoracic CT
• Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
• LoRA: Low-Rank Adaptation of Large Language Models
• Learning Transferable Visual Models From Natural Language Supervision
• Shortcut learning in deep neural networks
• Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study
• Representation Learning with Contrastive Predictive Coding
• Attention-based Deep Multiple Instance Learning
• On Calibration of Modern Neural Networks
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

Discussions

• In the Journal of Biomedical Informatics, @uclaradiology.bsky.social researchers evaluate a semantic-guided model linking CT nodule images with clinical descriptors. Results show strong malignancy pre [bsky, 1 points, 0 comments]

Related

• An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classification
• LungNoduleAgent: A Collaborative Multi-Agent System for Precision Diagnosis of Lung Nodules
• Highly accurate model for prediction of lung nodule malignancy with CT scans
• Learning from Ambiguous Labels for Lung Nodule Malignancy Prediction
• Faithful learning with sure data for lung nodule diagnosis