Pathology-and-genomics Multimodal Transformer for Survival Outcome Prediction

2023/07/22 by Kexin Ding, Ding, Kexin, Mu Zhou +5 · 5 citations
Computer Science · Medicine · #AI in cancer detection #Artificial Intelligence (cs.AI) #Colorectal Cancer Screening and Detection #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Radiomics and Machine Learning in Medical Imaging

paper · pdf · doi:10.48550/arxiv.2307.11952

openalex publication_date 2023/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Survival outcome assessment is challenging and inherently associated with multiple clinical factors (e.g., imaging and genomics biomarkers) in cancer. Enabling multimodal analytics promises to reveal novel predictive patterns of patient outcomes. In this study, we propose a multimodal transformer (PathOmics) integrating pathology and genomics insights into colon-related cancer survival prediction. We emphasize the unsupervised pretraining to capture the intrinsic interaction between tissue microenvironments in gigapixel whole slide images (WSIs) and a wide range of genomics data (e.g., mRNA-sequence, copy number variant, and methylation). After the multimodal knowledge aggregation in pretraining, our task-specific model finetuning could expand the scope of data utility applicable to both multi- and single-modal data (e.g., image- or genomics-only). We evaluate our approach on both TCGA colon and rectum cancer cohorts, showing that the proposed approach is competitive and outperforms state-of-the-art studies. Finally, our approach is desirable to utilize the limited number of finetuned samples towards data-efficient analytics for survival outcome prediction. The code is available at https://github.com/Cassie07/PathOmics.

Cited by

• Structural Prognostic Event Modeling for Multimodal Cancer Survival Analysis
• Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis
• Libra-MIL: Multimodal Prototypes Stereoscopic Infused with Task-specific Language Priors for Few-shot Whole Slide Image Classification
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research

Related

• MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features
• Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction
• Path-GPTOmic: A Balanced Multi-modal Learning Framework for Survival Outcome Prediction
• PS3: A Multimodal Transformer Integrating Pathology Reports with Histology Images and Biological Pathways for Cancer Survival Prediction
• Memory-Augmented Incomplete Multimodal Survival Prediction via Cross-Slide and Gene-Attentive Hypergraph Learning