2025/02/10 by Samuel Stevens, Stevens, Samuel, Wei-Lun Chao +7 · 1 voice · 14 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CV
paper · pdf · doi:10.48550/arxiv.2502.06755
openalex publication_date 2025/02/10 · arxiv published 2025/02/10 · openalex created_date 2025/10/10 · arxiv updated 2025/11/21 · openalex updated_date 2026/07/31
To truly understand vision models, we must not only interpret their learned features but also validate these interpretations through controlled experiments. While earlier work offers either rich semantics or direct control, few post-hoc tools supply both in a single, model-agnostic procedure. We use sparse autoencoders (SAEs) to bridge this gap; each sparse feature comes with real-image exemplars that reveal its meaning and a decoding vector that can be manipulated to probe its influence on downstream task behavior. By applying our method to widely-used pre-trained vision models, we reveal meaningful differences in the semantic abstractions learned by different pre-training objectives. We then show that a single SAE trained on frozen ViT activations supports patch-level causal edits across tasks (classification and segmentation) all without retraining the ViT or task heads. These qualitative, falsifiable demonstrations position SAEs as a practical bridge between concept discovery and causal probing of vision models. We provide code, demos and models on our project website: https://osu-nlp-group.github.io/saev.