2024/10/13 by George Crowley, Stephen R. Quake · 1 voice · 1 citation
Biochemistry, Genetics and Molecular Biology · #Cell Image Analysis Techniques #Gene Regulatory Network Analysis #Single-cell and spatial transcriptomics
paper · pdf · doi:10.1101/2024.10.10.617605
openalex publication_date 2024/10/13 · openalex created_date 2024/10/14 · openalex updated_date 2026/07/14
Abstract We developed an open-source package called AnnDictionary ( https://github.com/ggit12/anndictionary/ ) to facilitate the parallel, independent analysis of multiple anndata. AnnDictionary is built on top of LangChain and AnnData and supports all common large language model (LLM) providers. AnnDictionary only requires 1 line of code to configure or switch the LLM backend and it contains numerous multithreading optimizations to support the analysis of many anndata and large anndata. We used AnnDictionary to perform the first benchmarking study of all major LLMs at de novo cell-type annotation. LLMs varied greatly in absolute agreement with manual annotation based on model size. Inter-LLM agreement also varied with model size. We find that LLM annotation of most major cell types to be more than 80-90% accurate, and will maintain a leaderboard of LLM cell type annotation at https://singlecellgpt.com/celltype-annotation-leaderboard . Furthermore, we benchmarked these LLMs at functional annotation of gene sets, and found that Claude 3.5 Sonnet recovers close matches of functional gene set annotations in over 80% of test sets.