Multimodal contrastive learning for spatial gene expression prediction using histology images

2024/07/11 by Wenwen Min, Min, Wenwen, Zhiceng Shi +7 · 10 citations
Biochemistry, Genetics and Molecular Biology · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Biological sciences #FOS: Computer and information sciences #FOS: Electrical engineering #Gene expression and cancer classification #Image and Video Processing (eess.IV) #Quantitative Methods (q-bio.QM) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2407.08216

openalex publication_date 2024/07/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images (WSIs) stained with Hematoxylin and Eosin (H&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose mclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a "word", integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. Our extensive evaluation of mclSTExp on two breast cancer datasets and a skin squamous cell carcinoma dataset demonstrates its superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.

Cited by

• SCR2-ST: Combine Single Cell with Spatial Transcriptomics for Efficient Active Sampling via Reinforcement Learning
• Dual-Path Knowledge-Augmented Contrastive Alignment Network for Spatially Resolved Transcriptomics
• 3D-Guided Scalable Flow Matching for Generating Volumetric Tissue Spatial Transcriptomics from Serial Histology
• GenAR: Next-Scale Autoregressive Generation for Spatial Gene Expression Prediction
• PEaRL: Pathway-Enhanced Representation Learning for Gene and Pathway Expression Prediction from Histology
• Img2ST-Net: Efficient High-Resolution Spatial Omics Prediction from Whole Slide Histology Images via Fully Convolutional Image-to-Image Learning
• Gene-DML: Dual-Pathway Multi-Level Discrimination for Gene Expression Prediction from Histopathology Images
• SPATIA: Multimodal Generation and Prediction of Spatial Cell Phenotypes
• Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning
• Spatially Gene Expression Prediction using Dual-Scale Contrastive Learning

Related

• Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning
• Dual-Path Knowledge-Augmented Contrastive Alignment Network for Spatially Resolved Transcriptomics
• Spatially Resolved Gene Expression Prediction from Histology via Multi-view Graph Contrastive Learning with HSIC-bottleneck Regularization
• Spatially Gene Expression Prediction using Dual-Scale Contrastive Learning
• MMAP: A Multi-Magnification and Prototype-Aware Architecture for Predicting Spatial Gene Expression