What large language models know and what people think they know

2024/01/24 by Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar +7 · 1 voice · 150 citations
Computer Science · Medicine · Psychology · #Artificial Intelligence in Healthcare and Education #Artificial intelligence #Calibration #Cognitive psychology #Computer science #Explainable Artificial Intelligence (XAI) #Perception #Psychology #Self-confidence #Social psychology #Statistics #Topic Modeling

paper · pdf · doi:10.1038/s42256-024-00976-7

published in Nature Machine Intelligence 7(2), 221-231 (Nature Portfolio)

openalex publication_date 2025/01/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Abstract As artificial intelligence systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs’ internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models’ actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models’ internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in artificial-intelligence-assisted decision-making environments.

Citations

• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

Cited by

• Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking
• Variational Learning of Physical Intuition from a Few Observations: Charting Manifolds of Variational Physics
• Information Discernment in Large Language Models
• When AI Takes Sides on Questions of Faith: Persistent Asymmetries in AI-Mediated Faith Guidance
• Hallucinations Undermine Trust; Metacognition is a Way Forward
• Response drift across frontier large language models
• Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content
• Strategies for community-sourced biocuration in bioinformatics: a case study on MIBiG 4.0
• The Homogenizing Effect of Large Language Models on Human Expression and Thought
• Generative Deep Learning for de Novo Drug Design─A Chemical Space Odyssey
• Wrong and More Confident: A Field Experiment on Large Language Models Taking a Graduate Economics Exam
• Psychological Competence as a Missing Dimension in AI Evaluation
• Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error
• Metacognitive Sensitivity for Test-Time Dynamic Model Selection
• Calibrated Trust in Dealing with LLM Hallucinations: A Qualitative Study
• On the Measure of a Model: From Intelligence to Generality
• "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models
• Making LLMs Reliable When It Matters Most: A Five-Layer Architecture for High-Stakes Decisions
• How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
• ‘Appears to be about’: an evaluation of AI-generated metadata quality for community archives
• Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
• Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
• Teaching Language Models to Faithfully Express their Uncertainty
• How do LLMs Compute Verbal Confidence
• Quantifying uncert-AI-nty: Testing the accuracy of LLMs’ confidence judgments
• ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
• Improving Metacognition and Uncertainty Communication in Language Models
• When Large Language Models are Reliable for Judging Empathic Communication
• Exploring Similarity between Neural and LLM Trajectories in Language Processing
• Calibrating Verbalized Confidence with Self-Generated Distractors
• Self-reported perception of statistical literacy: Evidence from a National Survey of U.S. Adults
• Designing, Implementing, and Evaluating AI Explanations: A Scoping Review of Explainable AI Frameworks
• RELISH: LLM REgression with a Latent Iterative State Head
• LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
• Incorporating generative AI into a writing-intensive undergraduate course without off-loading learning
• The homogenizing effect of large language models on human expression and thought
• Metacognition and Uncertainty Communication in Humans and Large Language Models
• A closed-loop AI framework for hypothesis-driven and interpretable materials design
• Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design
• PIE: Performance Interval Estimation for Free-Form Generation Tasks
• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs
• Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
• Urbanite: A Dataflow-Based Framework for Human-AI Interactive Alignment in Urban Visual Analytics
• Enhancing Joint Human-AI Inference in Robot Missions: A Confidence-Based Approach
• Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
• Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing
• Hollow-LLM Attack: Computationally Trivial Weights in Zero-Knowledge Verification of LLM Inference
• How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models
• AI Agent Behavioral Science
• Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for Cybersecurity
• Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs
• Counterfactual Simulatability of LLM Explanations for Generation Tasks
• Explaining Sources of Uncertainty in Automated Fact-Checking
• MTRE: Multi-Token Reliability Estimation for Hallucination Detection in VLMs
• Metacognition in LLMs: Foundations, Progress, and Opportunities
• Online Safety Monitoring for LLMs
• Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents
• LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations
• Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review
• When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoption
• AI, Metacognition, and the Verification Bottleneck: A Three-Wave Longitudinal Study of Human Problem-Solving
• The Beginning of ChatGPT Ads
• A closer look at how large language models trust humans: patterns and biases
• Deep Learning in Concealed Dense Prediction
• Hallucination Detection on a Budget: Efficient Bayesian Estimation of Semantic Entropy

Discussions

• Cool stuff - engineering LLMs to produce low/high confidence statements based on internal probabilities increases users' alignment with actual LLM confidence (which for MCQs at least is pretty well ca [bsky, 29 points, 2 comments]

Related

• Large Language Models are overconfident and amplify human bias
• Improving Metacognition and Uncertainty Communication in Language Models
• Confidence in the Reasoning of Large Language Models
• Confronting verbalized uncertainty: Understanding how LLM’s verbalized uncertainty influences users in AI-assisted decision-making
• Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty