Towards Transparent AI: A Survey on Explainable Large Language Models

2025/06/26 by Avash Palikhe, Palikhe, Avash, Z Yu +5 · 3 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.21812

openalex publication_date 2025/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have played a pivotal role in advancing Artificial Intelligence (AI). However, despite their achievements, LLMs often struggle to explain their decision-making processes, making them a 'black box' and presenting a substantial challenge to explainability. This lack of transparency poses a significant obstacle to the adoption of LLMs in high-stakes domain applications, where interpretability is particularly essential. To overcome these limitations, researchers have developed various explainable artificial intelligence (XAI) methods that provide human-interpretable explanations for LLMs. However, a systematic understanding of these methods remains limited. To address this gap, this survey provides a comprehensive review of explainability techniques by categorizing XAI methods based on the underlying transformer architectures of LLMs: encoder-only, decoder-only, and encoder-decoder models. Then these techniques are examined in terms of their evaluation for assessing explainability, and the survey further explores how these explanations are leveraged in practical applications. Finally, it discusses available resources, ongoing research challenges, and future directions, aiming to guide continued efforts toward developing transparent and responsible LLMs.

Citations

• Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
• Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions
• XAI for All: Can Large Language Models Simplify Explainable AI?
• TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents
• Explainability for Large Language Models: A Survey
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Cognitive Network Science Reveals Bias in GPT-3, GPT-3.5 Turbo, and GPT-4 Mirroring Math Anxiety in High-School Students
• On the Reliability and Explainability of Language Models for Program Generation
• On the Reliability and Explainability of Language Models for Program Generation
• Progress measures for grokking via mechanistic interpretability
• Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
• In-context Learning and Induction Heads
• INTERACTION: A Generative XAI Framework for Natural Language Inference Explanations
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Can language models learn from explanations in context?
• Locating and Editing Factual Associations in GPT
• Attention Is Indeed All You Need: Semantically Attention-Guided Decoding\n for Data-to-Text NLG
• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• Implicit Representations of Meaning in Neural Language Models
• Discourse Probing of Pretrained Language Models
• Attention Head Masking for Inference Time Content Selection in Abstractive Summarization
• Exploring the Role of BERT Token Representations to Explain Sentence\n Probing Results
• On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
• Lite Training Strategies for Portuguese-English and English-Portuguese\n Translation
• Leveraging Passage Retrieval with Generative Models for Open Domain\n Question Answering
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language\n Generation, Translation, and Comprehension
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Designing and Interpreting Probes with Control Tasks
• On Identifiability in Transformers
• A Multiscale Visualization of Attention in the Transformer Model
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling
• Higher-order Coreference Resolution with Coarse-to-fine Inference
• A Unified Approach to Interpreting Model Predictions
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review
• CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
• Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies

Related

• Towards Transparent AI: A Survey on Explainable Language Models
• XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models
• LLMs for Explainable AI: A Comprehensive Survey
• Explainable artificial intelligence (XAI): from inherent explainability to large language models
• From Understanding to Utilization: A Survey on Explainability for Large Language Models