Zero-shot evaluation reveals limitations of single-cell foundation models

2025/04/18 by Kasia Z. Kedzierska, Lorin Crawford, Ava P. Amini +1 · 1 voice · 5 citations
Biochemistry, Genetics and Molecular Biology · #Advanced Fluorescence Microscopy Techniques #Cell Image Analysis Techniques #Single-cell and spatial transcriptomics

paper · pdf · doi:10.1186/s13059-025-03574-x

openalex publication_date 2025/04/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Foundation models such as scGPT and Geneformer have not been rigorously evaluated in a setting where they are used without any further training (i.e., zero-shot). Understanding the performance of models in zero-shot settings is critical to applications that exclude the ability to fine-tune, such as discovery settings where labels are unknown. Our evaluation of the zero-shot performance of Geneformer and scGPT suggests that, in some cases, these models may face reliability challenges and could be outperformed by simpler methods. Our findings underscore the importance of zero-shot evaluations in development and deployment of foundation models in single-cell research.

Citations

• Measuring Massive Multitask Language Understanding
• Zero-Shot Text-to-Image Generation
• Highly accurate protein structure prediction with AlphaFold
• On the Opportunities and Risks of Foundation Models
• Benchmarking atlas-level data integration in single-cell genomics
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Transfer learning enables predictions in network biology
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Cited by

• scPlantAnnotate: an accurate and robust transformer-based model for plant cell type annotation
• Limitations of cell embedding metrics assessed using drifting islands
• Learning Fragmentation Physics or Exploiting Sequence Priors? Benchmarking Bias in Deep Learning Models for De Novo Peptide Sequencing
• Interpretation, extrapolation and perturbation of single cells
• scE2TM improves single-cell embedding interpretability and reveals cellular perturbation signatures

Discussions

• 🖥️🧬 Foundation models scGPT and Geneformer promise universal single-cell embeddings, but zero-shot tests reveal they underperform simpler methods and flounder on batch effects. Improved pretraining [bsky, 2 points, 0 comments]

Related

• Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction
• Towards Applying Large Language Models to Complement Single-Cell Foundation Models
• Auditing pretraining contamination in single-cell foundation model benchmarks
• TEDDY: A Family Of Foundation Models For Understanding Single Cell Biology
• Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation