Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classification

2025/08/28 by Ayaka Tsutsumi, Guang Li, Tsutsumi, Ayaka +7
Computer Science · Engineering · Neuroscience · #AI in cancer detection #Artificial Intelligence (cs.AI) #Brain Tumor Detection and Classification #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Medical Imaging and Analysis

paper · pdf · doi:10.48550/arxiv.2508.20461

openalex publication_date 2025/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a novel medical image classification method that integrates dual-model weight selection with self-knowledge distillation (SKD). In real-world medical settings, deploying large-scale models is often limited by computational resource constraints, which pose significant challenges for their practical implementation. Thus, developing lightweight models that achieve comparable performance to large-scale models while maintaining computational efficiency is crucial. To address this, we employ a dual-model weight selection strategy that initializes two lightweight models with weights derived from a large pretrained model, enabling effective knowledge transfer. Next, SKD is applied to these selected models, allowing the use of a broad range of initial weight configurations without imposing additional excessive computational cost, followed by fine-tuning for the target classification tasks. By combining dual-model weight selection with self-knowledge distillation, our method overcomes the limitations of conventional approaches, which often fail to retain critical information in compact models. Extensive experiments on publicly available datasets-chest X-ray images, lung computed tomography scans, and brain magnetic resonance imaging scans-demonstrate the superior performance and robustness of our approach compared to existing methods.

Citations

• Continual Self-supervised Learning Considering Medical Domain Knowledge in Chest CT Images
• Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models
• Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
• Initializing Models with Larger Ones
• Mimetic Initialization of Self-Attention Layers
• Boosting Automatic COVID-19 Detection Performance with Self-Supervised Learning and Batch Knowledge Ensembling
• COVID-19 Detection Based on Self-Supervised Transfer Learning Using Chest X-Ray Images
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Understanding the Covariance Structure of Convolutional Filters
• A ConvNet for the 2020s
• A ConvNet for the 2020s
• MLP-Mixer: An all-MLP Architecture for Vision
• Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
• Self-Supervised Learning for Gastritis Detection with Gastric X-ray Images
• Transformer in Transformer
• Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Weight Distillation: Transferring the Knowledge in Neural Network Parameters
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Knowledge Distillation: A Survey
• Regularizing Class-wise Predictions via Self-knowledge Distillation
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Big Transfer (BiT): General Visual Representation Learning
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Contrastive Multiview Coding
• Contrastive Multiview Coding
• Class-incremental Learning via Deep Model Consolidation
• Improved Knowledge Distillation via Teacher Assistant
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Understanding deep learning requires rethinking generalization
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Distilling the Knowledge in a Neural Network
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• FitNets: Hints for Thin Deep Nets
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• Big Self-Supervised Models Advance Medical Image Classification
• Boosting Few-Shot Learning with Disentangled Self-Supervised Learning and Meta-Learning for Medical Image Classification
• Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification
• Dataset Distillation in Medical Imaging: A Feasibility Study
• Medical Knowledge-Guided Deep Learning for Imbalanced Medical Image Classification