CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders

2025/08/31 by Alex Gulko, Gulko, Alex, Yusen Peng +3
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2509.00691

openalex publication_date 2025/08/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sparse autoencoders (SAEs) are a promising approach for uncovering interpretable features in large language models (LLMs). While several automated evaluation methods exist for SAEs, most rely on external LLMs. In this work, we introduce CE-Bench, a novel and lightweight contrastive evaluation benchmark for sparse autoencoders, built on a curated dataset of contrastive story pairs. We conduct comprehensive evaluation studies to validate the effectiveness of our approach. Our results show that CE-Bench reliably measures the interpretability of sparse autoencoders and aligns well with existing benchmarks without requiring an external LLM judge, achieving over 70% Spearman correlation with results in SAEBench. The official implementation and evaluation dataset are open-sourced and publicly available.

Citations

• Persona Vectors: Monitoring and Controlling Character Traits in Language Models
• SAEs Are Good for Steering -- If You Select the Right Features
• Learning Multi-Level Features with Matryoshka Sparse Autoencoders
• SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
• Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
• Not All Language Model Features Are One-Dimensionally Linear
• Improving Dictionary Learning with Gated Sparse Autoencoders

Related

• SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
• Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
• Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders
• Evaluating SAE interpretability without explanations