Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks

2025/11/10 by Yucheng Zhou, Song, Lingran, Zhou, Yucheng +2
Computer Science · Medicine · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.06665

openalex publication_date 2025/11/10 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28

Abstract

Despite significant progress in pixel-level medical image analysis, existing medical image segmentation models rarely explore medical segmentation and diagnosis tasks jointly. However, it is crucial for patients that models can provide explainable diagnoses along with medical segmentation results. In this paper, we introduce a medical vision-language task named Medical Diagnosis Segmentation (MDS), which aims to understand clinical queries for medical images and generate the corresponding segmentation masks as well as diagnostic results. To facilitate this task, we first present the Multimodal Multi-disease Medical Diagnosis Segmentation (M3DS) dataset, containing diverse multimodal multi-disease medical images paired with their corresponding segmentation masks and diagnosis chain-of-thought, created via an automated diagnosis chain-of-thought generation pipeline. Moreover, we propose Sim4Seg, a novel framework that improves the performance of diagnosis segmentation by taking advantage of the Region-Aware Vision-Language Similarity to Mask (RVLS2M) module. To improve overall performance, we investigate a test-time scaling strategy for MDS tasks. Experimental results demonstrate that our method outperforms the baselines in both segmentation and diagnosis.

Citations

• MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration
• PRS-Med: Position Reasoning Segmentation with Vision-Language Model in Medical Imaging
• MediSee: Reasoning-based Pixel-level Perception in Medical Images
• HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
• Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback
• Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
• Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline
• LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
• Text4Seg: Reimagining Image Segmentation as Text Generation
• Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
• Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
• GLaMM: Pixel Grounding Large Multimodal Model
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
• A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation
• Learning Transferable Visual Models From Natural Language Supervision
• Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression
• Kvasir-SEG: A Segmented Polyp Dataset
• Generalized Intersection over Union: A Metric and A Loss for Bounding\n Box Regression
• nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• The Importance of Skip Connections in Biomedical Image Segmentation
• DeepCut: Object Segmentation from Bounding Box Annotations using\n Convolutional Neural Networks
• Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017\n International Symposium on Biomedical Imaging (ISBI), Hosted by the\n International Skin Imaging Collaboration (ISIC)
• Segmentation from Natural Language Expressions
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• A Closer Look at the Explainability of Contrastive Language-Image Pre-training

Related

• MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models
• RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology
• Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions
• MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairs
• SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer