TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

2021/06/02 by Zhuchen Shao, Hao Bian, Shao, Zhuchen +11 · 74 citations
Computer Science · #AI in cancer detection #Computer Vision and Pattern Recognition (cs.CV) #Digital Imaging for Blood Diseases #FOS: Computer and information sciences #Image Retrieval and Classification Techniques

paper · pdf · doi:10.48550/arxiv.2106.00908

openalex publication_date 2021/06/02 · openalex created_date 2021/11/22 · openalex updated_date 2026/07/28

Abstract

Multiple instance learning (MIL) is a powerful tool to solve the weakly supervised classification in whole slide image (WSI) based pathology diagnosis. However, the current MIL methods are usually based on independent and identical distribution hypothesis, thus neglect the correlation among different instances. To address this problem, we proposed a new framework, called correlated MIL, and provided a proof for convergence. Based on this framework, we devised a Transformer based MIL (TransMIL), which explored both morphological and spatial information. The proposed TransMIL can effectively deal with unbalanced/balanced and binary/multiple classification with great visualization and interpretability. We conducted various experiments for three different computational pathology problems and achieved better performance and faster convergence compared with state-of-the-art methods. The test AUC for the binary tumor classification can be up to 93.09% over CAMELYON16 dataset. And the AUC over the cancer subtypes classification can be up to 96.03% and 98.82% over TCGA-NSCLC dataset and TCGA-RCC dataset, respectively. Implementation is available at: https://github.com/szc19990412/TransMIL.

Cited by

• Weakly Supervised Instance-Level Gleason Pattern Estimation Using Primary and Secondary Labels
• PathSelect: Sequential Token Selection for Whole Slide Pathology
• Robustifying pathology foundation models via fine-tuning
• TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning
• WSD-MIL: Window Scale Decay Multiple Instance Learning for Whole Slide Image Classification
• DeltaMIL: Gated Memory Integration for Efficient and Discriminative Whole Slide Image Analysis
• MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Image
• Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosis
• A Multicenter Benchmark of Multiple Instance Learning Models for Lymphoma Subtyping from HE-stained Whole Slide Images
• LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
• CAPRMIL: Context-Aware Patch Representations for Multiple Instance Learning
• StainNet: Scaling Self-Supervised Foundation Models on Immunohistochemistry and Special Stains for Computational Pathology
• Cross-Stain Contrastive Learning for Paired Immunohistochemistry and Histopathology Slide Representation Learning
• milearn: A Python Package for Multi-Instance Machine Learning
• Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
• Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
• A Vector Symbolic Approach to Multiple Instance Learning
• nnMIL: A generalizable multiple instance learning framework for computational pathology
• MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
• ConSurv: Multimodal Continual Learning for Survival Analysis
• Positive Semi-definite Latent Factor Grouping-Boosted Cluster-reasoning Instance Disentangled Learning for WSI Representation
• Contrastive Integrated Gradients: A Feature Attribution-Based Method for Explaining Whole Slide Image Classification
• Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification
• Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
• Diagnose Like A REAL Pathologist: An Uncertainty-Focused Approach for Trustworthy Multi-Resolution Multiple Instance Learning
• Spatially-Aware Mixture of Experts with Log-Logistic Survival Modeling for Whole-Slide Images
• Dynamic Residual Encoding with Slide-Level Contrastive Learning for End-to-End Whole Slide Image Representation
• GAS-MIL: Group-Aggregative Selection Multi-Instance Learning for Ensemble of Foundation Models in Digital Pathology Image Analysis
• A multimodal whole-slide foundation model for pathology
• Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
• ITC-RWKV: Interactive Tissue-Cell Modeling with Recurrent Key-Value Aggregation for Histopathological Subtyping
• A Flow Model with Low-Rank Transformers for Incomplete Multimodal Survival Analysis
• ZACH-ViT: A Zero-Token Vision Transformer with ShuffleStrides Data Augmentation for Robust Lung Ultrasound Classification
• Morphology-Aware Prognostic model for Five-Year Survival Prediction in Colorectal Cancer from H&E Whole Slide Images
• Approximate Bilevel Graph Structure Learning for Histopathology Image Classification
• A Hierarchical Geometry-guided Transformer for Histological Subtyping of Primary Liver Cancer
• MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification
• DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction
• SlideMamba: Entropy-Based Adaptive Fusion of GNN and Mamba for Enhanced Representation Learning in Digital Pathology
• Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework
• C²MIL: Synchronizing Semantic and Topological Causalities in Multiple Instance Learning for Robust and Interpretable Survival Analysis
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning
• ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
• Multiple Instance Learning Framework with Masked Hard Instance Mining for Gigapixel Histopathology Image Analysis
• MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification
• CellEcoNet: Decoding the Cellular Language of Pathology with Deep Learning for Invasive Lung Adenocarcinoma Recurrence Prediction
• Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization
• Learning from Majority Label: A Novel Problem in Multi-class Multiple-Instance Learning
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• SemaMIL: Semantic-Aware Multiple Instance Learning with Retrieval-Guided State Space Modeling for Whole Slide Images
• Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtyping
• PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis
• The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology
• Controllable Latent Space Augmentation for Digital Pathology
• Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction
• IPGPhormer: Interpretable Pathology Graph-Transformer for Survival Analysis
• STAMP: Multi-pattern Attention-aware Multiple Instance Learning for STAS Diagnosis in Multi-center Histopathology Images
• Deformable Attention Graph Representation Learning for Histopathology Whole Slide Image Analysis
• Efficient Multi-Slide Visual-Language Feature Fusion for Placental Disease Classification
• Contrastive Cross-Bag Augmentation for Multiple Instance Learning-based Whole Slide Image Classification
• Welcome New Doctor: Continual Learning with Expert Consultation and Autoregressive Inference for Whole Slide Image Analysis
• Identifying actionable driver mutations in lung cancer using an efficient Asymmetric Transformer Decoder
• Minimal High-Resolution Patches Are Sufficient for Whole Slide Image Representation via Cascaded Dual-Scale Reconstruction
• Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment
• GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification
• Masked Omics Modeling for Multimodal Representation Learning across Histopathology and Molecular Profiles
• PTCMIL: Multiple Instance Learning via Prompt Token Clustering for Whole Slide Image Analysis
• Interpretable classification of pathology whole-slide images using attention based context-aware graph convolutional neural network
• Interpretable weakly-supervised learning through kernel density matrices: A digital pathology use case
• Survival Modeling from Whole Slide Images via Patch-Level Graph Clustering and Mixture Density Experts
• SGPMIL: Sparse Gaussian Process Multiple Instance Learning
• Single Domain Generalization for Multimodal Cross-Cancer Prognosis via Dirac Rebalancer and Distribution Entanglement
• Cracking Instance Jigsaw Puzzles: An Alternative to Multiple Instance Learning for Whole Slide Image Analysis

Related

• Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions
• The Whole Pathological Slide Classification via Weakly Supervised Learning
• TPMIL: Trainable Prototype Enhanced Multiple Instance Learning for Whole Slide Image Classification
• A Spatially-Aware Multiple Instance Learning Framework for Digital Pathology
• DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classification