LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization

2025/10/05 by Jiarui Liu, Liu, Jiarui, Joanna Jain +5 · 2 citations
Computer Science · Medicine · #Explainable Artificial Intelligence (XAI) #Artificial Intelligence in Healthcare and Education #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2510.04013

Abstract

Although large language models (LLMs) have tremendous utility, trustworthiness is still a chief concern: models often generate incorrect information with high confidence. While contextual information can help guide generation, identifying when a query would benefit from retrieved context and assessing the effectiveness of that context remains challenging. In this work, we operationalize interpretability methods to ascertain whether we can predict the correctness of model outputs from the model's activations alone. We also explore whether model internals contain signals about the efficacy of external context. We consider correct, incorrect, and irrelevant context and introduce metrics to distinguish amongst them. Experiments on six different models reveal that a simple classifier trained on intermediate layer activations of the first output token can predict output correctness with about 75% accuracy, enabling early auditing. Our model-internals-based metric significantly outperforms prompting baselines at distinguishing between correct and incorrect context, guarding against inaccuracies introduced by polluted context. These findings offer a lens to better understand the underlying decision-making processes of LLMs. Our code is publicly available at https://github.com/jiarui-liu/LLM-Microscope

Citations

• Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
• MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools
• Taming Knowledge Conflicts in Language Models
• HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
• Qwen2.5 Technical Report
• A Survey on LLM-as-a-Judge
• ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
• LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries
• ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence
• The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
• Backward Lens: Projecting Language Model Gradients into the Vocabulary Space
• INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
• A Survey of Confidence Estimation and Calibration in Large Language Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
• Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
• Steering Language Models With Activation Engineering
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Do Large Language Models Know What They Don't Know?
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
• Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• The Internal State of an LLM Knows When It's Lying
• Hallucinations in Large Multilingual Translation Models
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
• Discovering Latent Knowledge in Language Models Without Supervision
• Language Models (Mostly) Know What They Know
• Extracting Latent Steering Vectors from Pretrained Language Models
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
• Locating and Editing Factual Associations in GPT
• Retrieval Augmentation Reduces Hallucination in Conversation
• Reducing conversational agents' overconfidence through linguistic calibration
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent
• Measuring Massive Multitask Language Understanding
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Recipes for building an open-domain chatbot
• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• The Curious Case of Neural Text Degeneration
• Correcting Length Bias in Neural Machine Translation
• TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for\n Reading Comprehension

Cited by

• How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
• How do LLMs Compute Verbal Confidence
• Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
• LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals

Related

• Context Matters: Data-Efficient Augmentation of Large Language Models for Scientific Applications
• Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering
• Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility
• Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
• Context Engineering for Trustworthiness: Rescorla Wagner Steering Under Mixed and Inappropriate Contexts