Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators

2025/08/09 by Hyo Jin Do, Do, Hyo Jin, Rachel Ostrand +10 · 1 voice
Computer Science · Medicine · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Baseline (sea) #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Hallucinating #Human-Computer Interaction (cs.HC) #Readability #Style (visual arts) #Transparency (behavior) #Usability #cs.AI #cs.HC

paper · pdf · doi:10.48550/arxiv.2508.06846

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/08/09 · arxiv published 2025/08/09 · arxiv updated 2025/08/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Large language models (LLMs) are susceptible to generating inaccurate or false information, often referred to as "hallucinations" or "confabulations." While several technical advancements have been made to detect hallucinated content by assessing the factuality of the model's responses, there is still limited research on how to effectively communicate this information to users. To address this gap, we conducted two scenario-based experiments with a total of 208 participants to systematically compare the effects of various design strategies for communicating factuality scores by assessing participants' ratings of trust, ease in validating response accuracy, and preference. Our findings reveal that participants preferred and trusted a design in which all phrases within a response were color-coded based on factuality scores. Participants also found it easier to validate accuracy of the response in this style compared to a baseline with no style applied. Our study offers practical design guidelines for LLM application developers and designers, aimed at calibrating user trust, aligning with user preferences, and enhancing users' ability to scrutinize LLM outputs.

Citations

• From Overload to Insight: Scaffolding Creative Ideation through Structuring Inspiration
• One Does Not Simply Meme Alone: Evaluating Co-Creativity Between LLMs and Humans in the Generation of Humor
• FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
• AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
• Evaluating Verifiability in Generative Search Engines
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection
• QuestEval: Summarization Asks for Fact-based Evaluation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• On Faithfulness and Factuality in Abstractive Summarization
• BLEURT: Learning Robust Metrics for Text Generation
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Evaluating the Factual Consistency of Abstractive Text Summarization
• BERTScore: Evaluating Text Generation with BERT
• Fitting Linear Mixed-Effects Models using lme4
• Trust in Automation: Designing for Appropriate Reliance
• TRUE: Re-evaluating Factual Consistency Evaluation
• Critical period effects in second language learning: The influence of maturational state on the acquisition of English as a second language

Discussions

• 📄"Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators" 👉 arxiv.org/pdf/2508.06846 We found that highlighting every phrase in a response using a color scale for [bsky, 0 points, 0 comments]

Related

• Facilitating Human-LLM Collaboration through Factuality Scores and Source Attributions
• Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods
• A Debate-Driven Experiment on LLM Hallucinations and Accuracy
• Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
• FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation