The Importance of Prompt Tuning for Automated Neuron Explanations

2023/10/09 by Justin Lee, Lee, Justin, Tuomas Oikarinen +9 · 2 citations
Computer Science · Materials Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.06200

openalex publication_date 2023/10/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances have greatly increased the capabilities of large language models (LLMs), but our understanding of the models and their safety has not progressed as fast. In this paper we aim to understand LLMs deeper by studying their individual neurons. We build upon previous work showing large language models such as GPT-4 can be useful in explaining what each neuron in a language model does. Specifically, we analyze the effect of the prompt used to generate explanations and show that reformatting the explanation prompt in a more natural way can significantly improve neuron explanation quality and greatly reduce computational cost. We demonstrate the effects of our new prompts in three different ways, incorporating both automated and human evaluations.

Cited by

• Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization
• Evaluating Neuron Explanations: A Unified Framework with Sanity Checks

Related

• Rigorously Assessing Natural Language Explanations of Neurons
• Partially Rewriting a Transformer in Natural Language
• Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model
• Automatically Interpreting Millions of Features in Large Language Models
• N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models