PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis

2024/08/20 by Yan Wu, Wu, Yan, Esther Wershof +18 · 1 voice · 16 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #Cell Image Analysis Techniques #FOS: Biological sciences #FOS: Computer and information sciences #Genomics (q-bio.GN) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #q-bio.GN #stat.ML

paper · pdf · doi:10.48550/arxiv.2408.10609

openalex publication_date 2024/08/20 · arxiv published 2024/08/20 · openalex created_date 2025/10/10 · arxiv updated 2025/10/24 · openalex updated_date 2026/07/28

Abstract

We introduce a comprehensive framework for modeling single cell transcriptomic responses to perturbations, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and furthers the use of these models to simulate genetic and chemical screens for therapeutic discovery.

Cited by

• VCWorld: A Biological World Model for Virtual Cell Simulation
• Effects of Distance Metrics and Scaling on the Perturbation Discrimination Score
• Morphologically Intelligent Perturbation Prediction with FORM
• PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation
• PerturbMap: Cross-Context Transfer of Single-Cell Perturbation Responses
• Modeling Gene Expression Distributional Shifts for Unseen Genetic Perturbations
• Diversity by Design: Addressing Mode Collapse Improves scRNA-seq Perturbation Modeling on Well-Calibrated Metrics
• LLM-Guided Retrieval for Prediction of Molecular Perturbation Responses
• TxPert: Leveraging Biochemical Relationships for Out-of-Distribution Transcriptomic Perturbation Prediction
• Response Magnitude as a Dominant Signal for Held-Out CRISPRi Perturbation Effect Prediction
• Querying Counterfactuals on Tissue Graphs with Supervised Disentanglement
• HarmonyCell: Automating Single-Cell Perturbation Modeling under Semantic and Distribution Shifts
• CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation
• Benchmarking virtual cell models for in-the-wild perturbation response
• STRAND: Sequence-Conditioned Transport for Single-Cell Perturbations
• MapPFN: Learning Causal Perturbation Maps in Context

Discussions

• 2️⃣ ”PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis” arxiv.org/abs/2408.10609 [bsky, 0 points, 1 comments]

Related

• A Systematic Comparison of Single-Cell Perturbation Response Prediction Models
• Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all
• ScAPE: A lightweight multitask learning baseline method to predict transcriptomic responses to perturbations
• Predicting gene essentiality and drug response from perturbation screens in preclinical cancer models with LEAP: Layered Ensemble of Autoencoders and Predictors
• Benchmarking virtual cell models for in-the-wild perturbation response