Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants

2025/12/17 by Vincent Huang, Huang, Vincent, Dami Choi +7 · 1 voice · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2512.15712

openalex publication_date 2025/12/17 · arxiv published 2025/12/17 · arxiv updated 2025/12/17 · openalex created_date 2025/12/19 · openalex updated_date 2026/07/28

Abstract

Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interpretability use hand-designed agents that make and test hypotheses about how internal activations relate to external behavior. We propose to instead turn this task into an end-to-end training objective, by training interpretability assistants to accurately predict model behavior from activations through a communication bottleneck. Specifically, an encoder compresses activations to a sparse list of concepts, and a decoder reads this list and answers a natural language question about the model. We show how to pretrain this assistant on large unstructured data, then finetune it to answer questions. The resulting architecture, which we call a Predictive Concept Decoder, enjoys favorable scaling properties: the auto-interp score of the bottleneck concepts improves with data, as does the performance on downstream applications. Specifically, PCDs can detect jailbreaks, secret hints, and implanted latent concepts, and are able to accurately surface latent user attributes.

Citations

• Reasoning Models Don't Always Say What They Think
• AssistanceZero: Scalably Solving Assistance Games
• Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
• HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworks
• LatentQA: Teaching LLMs to Decode Activations Into Natural Language
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
• Concept Bottleneck Models Without Predefined Concepts
• Semi-supervised Concept Bottleneck Models
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Scaling and evaluating sparse autoencoders
• Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
• A Multimodal Automated Interpretability Agent
• A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Measuring Faithfulness in Chain-of-Thought Reasoning
• Jailbroken: How Does LLM Safety Training Fail?
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• Natural Language Descriptions of Deep Visual Features
• Probing Classifiers: Promises, Shortcomings, and Advances
• Concept Bottleneck Models
• Understanding intermediate layers using linear classifier probes
• Cooperative Inverse Reinforcement Learning

Cited by

• Detecting the Disturbance: A Nuanced View of Introspective Abilities in LLMs
• Rep2Text: Decoding Full Text from a Single LLM Token Representation
• Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision

Discussions

• 5/ Predictive Concept Decoders train an end-to-end interp assistant: compress activations into concept lists, then answer questions about model behavior. X: https://x.com/damichoi95/status/20750129310 [bsky, 0 points, 1 comments]

Related

• Explanation Bottleneck Models
• Concept Bottleneck Model with Additional Unsupervised Concepts
• Interpretable Concept-Based Memory Reasoning
• Interpreting Pretrained Language Models via Concept Bottlenecks
• Decoder Tuning: Efficient Language Understanding as Decoding