Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking

2025/01/30 by Kevin Roitero, Dustin Wright, Michael Soprano +2 · 1 voice · 1 citation
Social Sciences · Computer Science · #Misinformation and Its Impacts #Topic Modeling #Spam and Phishing Detection

paper · pdf · doi:10.1145/3726302.3729960

Abstract

Evaluating the truthfulness of online content is critical for combating misinformation. This study examines the efficiency and effectiveness of crowdsourced truthfulness assessments through a comparative analysis of two approaches: one involving full-length webpages as evidence for each claim, and another using summaries for each evidence document generated with an LLM. Using an A/B testing setting, we engage a diverse pool of participants tasked with evaluating the truthfulness of statements under these conditions.

Citations

• On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
• Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking
• The Psychology of Fake News

Cited by

• The Effect of Document Summarization on LLM-Based Relevance Judgments

Discussions

• 📄 New preprint: "Collecting Cost-Effective, High-Quality Truthfulness Assessments with LLM Summarized Evidence" We show: fact checking w/ crowd workers is more efficient when using LLM summaries, qu [bsky, 19 points, 1 comments]

Related

• Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking
• In Crowd Veritas: Leveraging Human Intelligence To Fight Misinformation
• Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs
• Generating Fact Checking Briefs
• AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X