A graph-transformer for whole slide image classification

2022/05/19 by Yi Zheng, Zheng, Yi, Rushin H. Gindra +14 · 22 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #AI in cancer detection #Bioinformatics and Genomic Networks #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Gene expression and cancer classification #cs.CV

paper · pdf · doi:10.48550/arxiv.2205.09671

arxiv created 2022/05/19 · openalex publication_date 2022/05/19 · arxiv updated 2022/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy= 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.

Cited by

• PathSelect: Sequential Token Selection for Whole Slide Pathology
• A multimodal whole-slide foundation model for pathology
• A Hierarchical Geometry-guided Transformer for Histological Subtyping of Primary Liver Cancer
• MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification
• torchmil: A PyTorch-based library for deep Multiple Instance Learning
• SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification
• Standardized Multi-Layer Tissue Maps for Enhanced Artificial Intelligence Integration and Search in Large-Scale Whole Slide Image Archives
• IPGPhormer: Interpretable Pathology Graph-Transformer for Survival Analysis
• MOC: Meta-Optimized Classifier for Few-Shot Whole Slide Image Classification
• Contrastive Cross-Bag Augmentation for Multiple Instance Learning-based Whole Slide Image Classification
• Identifying actionable driver mutations in lung cancer using an efficient Asymmetric Transformer Decoder
• PTCMIL: Multiple Instance Learning via Prompt Token Clustering for Whole Slide Image Analysis
• Cracking Instance Jigsaw Puzzles: An Alternative to Multiple Instance Learning for Whole Slide Image Analysis
• MedFormer: Hierarchical Medical Vision Transformer with Content-Aware Dual Sparse Selection Attention
• Continual Multiple Instance Learning with Enhanced Localization for Histopathological Whole Slide Image Analysis
• Reducing Variability of Multiple Instance Learning Methods for Digital Pathology
• Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
• Histo-Miner: Deep Learning based Tissue Features Extraction Pipeline from H&E Whole Slide Images of Cutaneous Squamous Cell Carcinoma
• Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
• Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining
• From Patches to Evidence Balls: Class-Conditioned Evidence Retrieval for Few-Shot Whole Slide Image Classification
• Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
• Predicting ulcer in H&E images of inflammatory bowel disease using domain-knowledge-driven graph neural network

Related

• A Graph-Based Framework for Interpretable Whole Slide Image Analysis
• Representation Learning of Histopathology Images using Graph Neural\n Networks
• Context-Aware Self-Supervised Learning of Whole Slide Images
• Artifact-Robust Graph-Based Learning in Digital Pathology
• MulGT: Multi-task Graph-Transformer with Task-aware Knowledge Injection and Domain Knowledge-driven Pooling for Whole Slide Image Analysis