MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding

2025/06/10 by Shivang Chopra, Gabriela Sánchez-Rodríguez, Chopra, Shivang +9 · 3 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.08356

openalex publication_date 2025/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Different medical imaging modalities capture diagnostic information at varying spatial resolutions, from coarse global patterns to fine-grained localized structures. However, most existing vision-language frameworks in the medical domain apply a uniform strategy for local feature extraction, overlooking the modality-specific demands. In this work, we present MedMoE, a modular and extensible vision-language processing framework that dynamically adapts visual representation based on the diagnostic context. MedMoE incorporates a Mixture-of-Experts (MoE) module conditioned on the report type, which routes multi-scale image features through specialized expert branches trained to capture modality-specific visual semantics. These experts operate over feature pyramids derived from a Swin Transformer backbone, enabling spatially adaptive attention to clinically relevant regions. This framework produces localized visual representations aligned with textual descriptions, without requiring modality-specific supervision at inference. Empirical results on diverse medical benchmarks demonstrate that MedMoE improves alignment and retrieval performance across imaging modalities, underscoring the value of modality-specialized visual representations in clinical vision-language systems.

Citations

• UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for\n Diverse Medical Imaging Modalities
• MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
• Quilt-1M: One Million Image-Text Pairs for Histopathology
• PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• Advancing Radiograph Representation Learning with Masked Record Modeling
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
• Mixture-of-Experts with Expert Choice Routing
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• Learning Transferable Visual Models From Natural Language Supervision
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and\n Expert Comparison
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Cited by

• REN: Anatomically-Informed Mixture-of-Experts for Interstitial Lung Disease Diagnosis
• Mixture of Experts in Large Language Models
• MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification

Related

• MoME: Mixture of Visual Language Medical Experts for Medical Imaging Segmentation
• Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
• VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge
• MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models