2024/04/23 by Furui Cheng, Vilém Zouhar, Cheng, Furui +9 · 2 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Law #Computation and Language (cs.CL) #Digital Rights Management and Security #FOS: Computer and information sciences #H.5.2 #Human-Computer Interaction (cs.HC) #I.2.7 #Machine Learning (cs.LG) #Semantic Web and Ontologies
paper · pdf · doi:10.48550/arxiv.2405.00708
openalex publication_date 2024/04/23 · openalex created_date 2024/05/05 · openalex updated_date 2026/07/28
Understanding the behavior of large language models (LLMs) is crucial for ensuring their safe and reliable use. However, existing explainable AI (XAI) methods for LLMs primarily rely on word-level explanations, which are often computationally inefficient and misaligned with human reasoning processes. Moreover, these methods often treat explanation as a one-time output, overlooking its inherently interactive and iterative nature. In this paper, we present LLM Analyzer, an interactive visualization system that addresses these limitations by enabling intuitive and efficient exploration of LLM behaviors through counterfactual analysis. Our system features a novel algorithm that generates fluent and semantically meaningful counterfactuals via targeted removal and replacement operations at user-defined levels of granularity. These counterfactuals are used to compute feature attribution scores, which are then integrated with concrete examples in a table-based visualization, supporting dynamic analysis of model behavior. A user study with LLM practitioners and interviews with experts demonstrate the system's usability and effectiveness, emphasizing the importance of involving humans in the explanation process as active participants rather than passive recipients.