Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

2023/11/15 by Yuxia Wang, Wang, Yuxia, Revanth Gangi Reddy +23 · 34 citations
Decision Sciences · Computer Science · #Scientific Computing and Data Management #Data Quality and Management #Advanced Data Storage Technologies

paper · pdf · doi:10.48550/arxiv.2311.09000

Abstract

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.

Cited by

• FaithLens: Detecting and Explaining Faithfulness Hallucination
• RAG System for Supporting Japanese Litigation Procedures: Faithful Response Generation Complying with Legal Norms
• Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphs
• A Neurosymbolic Approach to Natural Language Formalization and Verification
• Stress Testing Factual Consistency Metrics for Long-Document Summarization
• DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queries
• MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
• Assessing Web Search Credibility and Response Groundedness in Chat Assistants
• Multi-stage Prompt Refinement for Mitigating Hallucinations in Large Language Models
• FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
• Exposing Citation Vulnerabilities in Generative Engines
• HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation
• Towards Ecologically Valid LLM Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
• MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responses
• Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
• LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification
• The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
• Paladin-mini: A Compact and Efficient Grounding Model Excelling in Real-World Scenarios
• Correcting Hallucinations in News Summaries: Exploration of Self-Correcting LLM Methods with External Knowledge
• RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking
• Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts
• The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
• Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs
• HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs
• Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation
• Explaining Sources of Uncertainty in Automated Fact-Checking
• VeriFastScore: Speeding up long-form factuality evaluation
• EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actions
• UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
• Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Ability
• Averaging Bias: Human Faithfulness Annotations are not Locally Faithful
• VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts
• ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models
• TriQua: Reconciling Granularity and Context in Factuality Evaluation
• PASS-FC: Progressive and Adaptive Search Scheme for Fact Checking of Comprehensive Claims

Related

• OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs
• FactLens: Benchmarking Fine-Grained Fact Verification
• FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
• OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs
• Generating Benchmarks for Factuality Evaluation of Language Models