Transcoder-based Circuit Analysis for Interpretable Single-Cell Foundation Models

2025/09/18 by Hosokawa, Sosuke, Kawakami, Toshiharu, Kodera, Satoshi +2
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2509.14723

Abstract

Single-cell foundation models (scFMs) have demonstrated state-of-the-art performance on various tasks, such as cell-type annotation and perturbation response prediction, by learning gene regulatory networks from large-scale transcriptome data. However, a significant challenge remains: the decision-making processes of these models are less interpretable compared to traditional methods like differential gene expression analysis. Recently, transcoders have emerged as a promising approach for extracting interpretable decision circuits from large language models (LLMs). In this work, we train a transcoder on the cell2sentence (C2S) model, a state-of-the-art scFM. By leveraging the trained transcoder, we extract internal decision-making circuits from the C2S model. We demonstrate that the discovered circuits correspond to real-world biological mechanisms, confirming the potential of transcoders to uncover biologically plausible pathways within complex single-cell models.

Citations

• Can sparse autoencoders make sense of gene expression latent variable models?
• Transcoders Find Interpretable LLM Feature Circuits
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Transfer learning enables predictions in network biology
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Governance Architecture for Neural Network Superposition: A Structural Solution to Hallucination via Routing and Interference Filtering

Related

• Deciphering Cell Systems: Machine Learning Perspectives And Approaches For The Analysis Of Single-Cell Data
• Cell-ontology guided transcriptome foundation model
• Cell2Text: Multimodal LLM for Generating Single-Cell Descriptions from RNA-Seq Data
• BMFM-RNA: whole-cell expression decoding improves transcriptomic foundation models
• Multi-modal single-cell foundation models via dynamic token adaptation