MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation

2025/09/07 by Yiwen Ye, Yicheng Wu, Ye, Yiwen +13
Computer Science · Engineering · Medicine · #3D Shape Modeling and Analysis #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Medical Image Segmentation Techniques #Radiomics and Machine Learning in Medical Imaging

paper · pdf · doi:10.48550/arxiv.2509.06096

openalex publication_date 2025/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Foundation models have become a promising paradigm for advancing medical image analysis, particularly for segmentation tasks where downstream applications often emerge sequentially. Existing fine-tuning strategies, however, remain limited: parallel fine-tuning isolates tasks and fails to exploit shared knowledge, while multi-task fine-tuning requires simultaneous access to all datasets and struggles with incremental task integration. To address these challenges, we propose MedSeqFT, a sequential fine-tuning framework that progressively adapts pre-trained models to new tasks while refining their representational capacity. MedSeqFT introduces two core components: (1) Maximum Data Similarity (MDS) selection, which identifies downstream samples most representative of the original pre-training distribution to preserve general knowledge, and (2) Knowledge and Generalization Retention Fine-Tuning (K&G RFT), a LoRA-based knowledge distillation scheme that balances task-specific adaptation with the retention of pre-trained knowledge. Extensive experiments on two multi-task datasets covering ten 3D segmentation tasks demonstrate that MedSeqFT consistently outperforms state-of-the-art fine-tuning strategies, yielding substantial performance gains (e.g., an average Dice improvement of 3.0%). Furthermore, evaluations on two unseen tasks (COVID-19-20 and Kidney) verify that MedSeqFT enhances transferability, particularly for tumor segmentation. Visual analyses of loss landscapes and parameter variations further highlight the robustness of MedSeqFT. These results establish sequential fine-tuning as an effective, knowledge-retentive paradigm for adapting foundation models to evolving clinical tasks. Code will be released.

Citations

• DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations
• Large-Scale Multi-Center CT and MRI Segmentation of Pancreas with Deep Learning
• Large-Scale 3D Medical Image Pre-training with Geometric Context Priors
• MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model
• SG-MIM: Structured Knowledge Guided Efficient Pre-training for Dense Prediction
• 5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning
• Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification
• Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning
• Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing
• AC-Norm: Effective Tuning for Medical Image Analysis via Affine Collaborative Normalization
• AlignDet: Aligning Pre-training and Fine-tuning in Object Detection
• Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation
• Med-Tuning: A New Parameter-Efficient Tuning Framework for Medical Volumetric Segmentation
• UniSeg: A Prompt-driven Universal Segmentation Model as well as A Strong Representation Learner
• Medical Image Segmentation Review: The success of U-Net
• Medical Image Segmentation Review: The Success of U-Net
• Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets
• DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysis
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis
• Masked Autoencoders Are Scalable Vision Learners
• LoRA: Low-Rank Adaptation of Large Language Models
• nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
• Efficient Conditional Pre-training for Transfer Learning
• DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets
• The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge
• Parameter-Efficient Transfer Learning for NLP
• Visualizing the Loss Landscape of Neural Nets

Related

• Task-Specific Knowledge Distillation from the Vision Foundation Model for Enhanced Medical Image Segmentation
• Multi-Prompt Fine-Tuning of Foundation Models for Enhanced Medical Image Segmentation
• Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially
• Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions
• MedicoSAM: Robust Improvement of SAM for Medical Imaging