2024/11/24 by Gustav Müller‐Franzes, Firas Khader, Müller-Franzes, Gustav +11 · 9 citations
Engineering · Medicine · Neuroscience · #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Medical Imaging and Analysis #Radiomics and Machine Learning in Medical Imaging #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2411.15802
openalex publication_date 2024/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
MRI and CT are essential clinical cross-sectional imaging techniques for diagnosing complex conditions. However, large 3D datasets with annotations for deep learning are scarce. While methods like DINOv2 are encouraging for 2D image analysis, these methods have not been applied to 3D medical images. Furthermore, deep learning models often lack explainability due to their "black-box" nature. This study aims to extend 2D self-supervised models, specifically DINOv2, to 3D medical imaging while evaluating their potential for explainable outcomes. We introduce the Medical Slice Transformer (MST) framework to adapt 2D self-supervised models for 3D medical image analysis. MST combines a Transformer architecture with a 2D feature extractor, i.e., DINOv2. We evaluate its diagnostic performance against a 3D convolutional neural network (3D ResNet) across three clinical datasets: breast MRI (651 patients), chest CT (722 patients), and knee MRI (1199 patients). Both methods were tested for diagnosing breast cancer, predicting lung nodule dignity, and detecting meniscus tears. Diagnostic performance was assessed by calculating the Area Under the Receiver Operating Characteristic Curve (AUC). Explainability was evaluated through a radiologist's qualitative comparison of saliency maps based on slice and lesion correctness. P-values were calculated using Delong's test. MST achieved higher AUC values compared to ResNet across all three datasets: breast (0.94±0.01 vs. 0.91±0.02, P=0.02), chest (0.95±0.01 vs. 0.92±0.02, P=0.13), and knee (0.85±0.04 vs. 0.69±0.05, P=0.001). Saliency maps were consistently more precise and anatomically correct for MST than for ResNet. Self-supervised 2D models like DINOv2 can be effectively adapted for 3D medical imaging using MST, offering enhanced diagnostic accuracy and explainability compared to convolutional neural networks.