TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

2021/06/02 by Zhuchen Shao, Hao Bian, Shao, Zhuchen +11 · 504 citations
Computer Science · #AI in cancer detection #Artificial intelligence #Binary classification #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Data mining #Digital Imaging for Blood Diseases #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Interpretability #Machine learning #Pattern recognition (psychology) #Support vector machine #Visualization #cs.CV

paper · pdf · doi:10.48550/arxiv.2106.00908

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/06/02 · arxiv created 2021/10/31 · arxiv updated 2021/11/02 · openalex created_date 2021/11/22 · openalex updated_date 2026/08/04

Abstract

Multiple instance learning (MIL) is a powerful tool to solve the weakly supervised classification in whole slide image (WSI) based pathology diagnosis. However, the current MIL methods are usually based on independent and identical distribution hypothesis, thus neglect the correlation among different instances. To address this problem, we proposed a new framework, called correlated MIL, and provided a proof for convergence. Based on this framework, we devised a Transformer based MIL (TransMIL), which explored both morphological and spatial information. The proposed TransMIL can effectively deal with unbalanced/balanced and binary/multiple classification with great visualization and interpretability. We conducted various experiments for three different computational pathology problems and achieved better performance and faster convergence compared with state-of-the-art methods. The test AUC for the binary tumor classification can be up to 93.09% over CAMELYON16 dataset. And the AUC over the cancer subtypes classification can be up to 96.03% and 98.82% over TCGA-NSCLC dataset and TCGA-RCC dataset, respectively. Implementation is available at: https://github.com/szc19990412/TransMIL.

Cited by

• Weakly Supervised Instance-Level Gleason Pattern Estimation Using Primary and Secondary Labels
• PathSelect: Sequential Token Selection for Whole Slide Pathology
• Robustifying pathology foundation models via fine-tuning
• TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learning
• WSD-MIL: Window Scale Decay Multiple Instance Learning for Whole Slide Image Classification
• DeltaMIL: Gated Memory Integration for Efficient and Discriminative Whole Slide Image Analysis
• MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Image
• Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosis
• A Multicenter Benchmark of Multiple Instance Learning Models for Lymphoma Subtyping from HE-stained Whole Slide Images
• LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
• CAPRMIL: Context-Aware Patch Representations for Multiple Instance Learning
• StainNet: Scaling Self-Supervised Foundation Models on Immunohistochemistry and Special Stains for Computational Pathology
• Cross-Stain Contrastive Learning for Paired Immunohistochemistry and Histopathology Slide Representation Learning
• milearn: A Python Package for Multi-Instance Machine Learning
• Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
• Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics
• A Vector Symbolic Approach to Multiple Instance Learning
• nnMIL: A generalizable multiple instance learning framework for computational pathology
• MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
• ConSurv: Multimodal Continual Learning for Survival Analysis
• Positive Semi-definite Latent Factor Grouping-Boosted Cluster-reasoning Instance Disentangled Learning for WSI Representation
• Contrastive Integrated Gradients: A Feature Attribution-Based Method for Explaining Whole Slide Image Classification
• Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification
• Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
• Diagnose Like A REAL Pathologist: An Uncertainty-Focused Approach for Trustworthy Multi-Resolution Multiple Instance Learning
• Spatially-Aware Mixture of Experts with Log-Logistic Survival Modeling for Whole-Slide Images
• Dynamic Residual Encoding with Slide-Level Contrastive Learning for End-to-End Whole Slide Image Representation
• GAS-MIL: Group-Aggregative Selection Multi-Instance Learning for Ensemble of Foundation Models in Digital Pathology Image Analysis
• A multimodal whole-slide foundation model for pathology
• Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
• ITC-RWKV: Interactive Tissue-Cell Modeling with Recurrent Key-Value Aggregation for Histopathological Subtyping
• A Flow Model with Low-Rank Transformers for Incomplete Multimodal Survival Analysis
• ZACH-ViT: A Zero-Token Vision Transformer with ShuffleStrides Data Augmentation for Robust Lung Ultrasound Classification
• Morphology-Aware Prognostic model for Five-Year Survival Prediction in Colorectal Cancer from H&E Whole Slide Images
• Approximate Bilevel Graph Structure Learning for Histopathology Image Classification
• A Hierarchical Geometry-guided Transformer for Histological Subtyping of Primary Liver Cancer
• MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification
• DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction
• SlideMamba: Entropy-Based Adaptive Fusion of GNN and Mamba for Enhanced Representation Learning in Digital Pathology
• Revisiting Data Challenges of Computational Pathology: A Pack-based Multiple Instance Learning Training Framework
• C²MIL: Synchronizing Semantic and Topological Causalities in Multiple Instance Learning for Robust and Interpretable Survival Analysis
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning
• ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
• Multiple Instance Learning Framework with Masked Hard Instance Mining for Gigapixel Histopathology Image Analysis
• MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification
• CellEcoNet: Decoding the Cellular Language of Pathology with Deep Learning for Invasive Lung Adenocarcinoma Recurrence Prediction
• Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization
• Learning from Majority Label: A Novel Problem in Multi-class Multiple-Instance Learning
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• SemaMIL: Semantic-Aware Multiple Instance Learning with Retrieval-Guided State Space Modeling for Whole Slide Images
• Boosting Pathology Foundation Models via Few-shot Prompt-tuning for Rare Cancer Subtyping
• PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis
• The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology
• Controllable Latent Space Augmentation for Digital Pathology
• Cross-Cancer Knowledge Transfer in WSI-based Prognosis Prediction
• IPGPhormer: Interpretable Pathology Graph-Transformer for Survival Analysis
• STAMP: Multi-pattern Attention-aware Multiple Instance Learning for STAS Diagnosis in Multi-center Histopathology Images
• Deformable Attention Graph Representation Learning for Histopathology Whole Slide Image Analysis
• Efficient Multi-Slide Visual-Language Feature Fusion for Placental Disease Classification
• Contrastive Cross-Bag Augmentation for Multiple Instance Learning-based Whole Slide Image Classification
• Welcome New Doctor: Continual Learning with Expert Consultation and Autoregressive Inference for Whole Slide Image Analysis
• Identifying actionable driver mutations in lung cancer using an efficient Asymmetric Transformer Decoder
• Minimal High-Resolution Patches Are Sufficient for Whole Slide Image Representation via Cascaded Dual-Scale Reconstruction
• Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment
• GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification
• Masked Omics Modeling for Multimodal Representation Learning across Histopathology and Molecular Profiles
• PTCMIL: Multiple Instance Learning via Prompt Token Clustering for Whole Slide Image Analysis
• Interpretable classification of pathology whole-slide images using attention based context-aware graph convolutional neural network
• Interpretable weakly-supervised learning through kernel density matrices: A digital pathology use case
• Survival Modeling from Whole Slide Images via Patch-Level Graph Clustering and Mixture Density Experts
• SGPMIL: Sparse Gaussian Process Multiple Instance Learning
• Single Domain Generalization for Multimodal Cross-Cancer Prognosis via Dirac Rebalancer and Distribution Entanglement
• Cracking Instance Jigsaw Puzzles: An Alternative to Multiple Instance Learning for Whole Slide Image Analysis
• GNN-ViTCap: GNN-Enhanced Multiple Instance Learning with Vision Transformers for Whole Slide Image Classification and Captioning
• Mammo-Clustering: Context Clustering based Multi-view Tri Level Information Fusion for Lesion Location and Classification in Mammography
• Classification of autoimmune diseases from Peripheral blood TCR repertoires by multimodal multi-instance learning
• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction
• MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attention
• Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image Analysis
• SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space
• Genome-Anchored Foundation Model Embeddings Improve Molecular Prediction from Histology Images
• FMaMIL: Frequency-Driven Mamba Multi-Instance Learning for Weakly Supervised Lesion Segmentation in Medical Images
• Transforming H&E images into IHC: A Variance-Penalized GAN for Precision Oncology
• DuoFormer: Leveraging Hierarchical Representations by Local and Global Attention Vision Transformer
• Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
• Histo-Miner: Deep Learning based Tissue Features Extraction Pipeline from H&E Whole Slide Images of Cutaneous Squamous Cell Carcinoma
• Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning
• PathGene: Benchmarking Driver Gene Mutations and Exon Prediction Using Multicenter Lung Cancer Histopathology Image Dataset
• The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Models
• DLiPath: A Benchmark for the Comprehensive Assessment of Donor Liver Based on Histopathological Image Dataset
• DSAGL: Dual-Stream Attention-Guided Learning for Weakly Supervised Whole Slide Image Classification
• Multi-instance Learning as Downstream Task of Self-Supervised Learning-based Pre-trained Model
• Hypergraph Mamba for Efficient Whole Slide Image Understanding
• Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
• Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification
• Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining
• From Patches to Evidence Balls: Class-Conditioned Evidence Retrieval for Few-Shot Whole Slide Image Classification
• Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
• Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance
• Advancing Multiple Instance Learning with Continual Learning for Whole Slide Imaging
• VIGIL: Vision-Language Guided Multiple Instance Learning Framework for Ulcerative Colitis Histological Healing Prediction
• What Carries the Signal in Pathology Foundation-Model Atlases? A Patient-Level Controlled Benchmark in Breast Cancer
• SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Images
• Lifelong Whole Slide Image Analysis: Online Vision-Language Adaptation and Past-to-Present Gradient Distillation
• Self-Supervision Enhances Instance-based Multiple Instance Learning Methods in Digital Pathology: A Benchmark Study
• A Deployment-Friendly Foundational Framework for Efficient Computational Pathology
• A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification
• Spatial proteomics guided by H&E-based AI reveals recurrence-risk niches in triple-negative breast cancer
• CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment
• A Spatially-Aware Multiple Instance Learning Framework for Digital Pathology
• A Multi-Cohort Validation of Censoring-Aware Conformal Lower Predictive Bounds for Pathology Survival Models
• ZeroSlide: Is Zero-Shot Classification Adequate for Lifelong Learning in Whole-Slide Image Analysis in the Era of Pathology Vision-Language Foundation Models?
• How Effective Can Dropout Be in Multiple Instance Learning ?
• Weakly Semi-supervised Whole Slide Image Classification by Two-level Cross Consistency Supervision
• Predicting ulcer in H&E images of inflammatory bowel disease using domain-knowledge-driven graph neural network

Related

• Advances in Multiple Instance Learning for Whole Slide Image Analysis: Techniques, Challenges, and Future Directions
• The Whole Pathological Slide Classification via Weakly Supervised Learning
• TPMIL: Trainable Prototype Enhanced Multiple Instance Learning for Whole Slide Image Classification
• A Spatially-Aware Multiple Instance Learning Framework for Digital Pathology
• DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classification