Long-Form Information Alignment Evaluation Beyond Atomic Facts

2025/05/21 by Danna Zheng, Mirella Lapata, Zheng, Danna +3 · 1 citation
Materials Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science

paper · pdf · doi:10.48550/arxiv.2505.15792

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.

Citations

• Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst
• Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation
• Atomic Fact Decomposition Helps Attributed Question Answering
• Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA
• CHECKWHY: Causal Fact Verification via Argument Structure
• FineSurE: Fine-grained Summarization Evaluation using LLMs
• VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation
• Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts
• MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
• TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization
• TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness
• Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
• CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance
• EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification
• AlignScore: Evaluating Factual Consistency with a Unified Alignment Function
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web
• Evaluating Verifiability in Generative Search Engines
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• ChatGPT as a Factual Inconsistency Evaluator for Text Summarization
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
• Knowledge Graph Reasoning with Logics and Embeddings: Survey and Perspective
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
• SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
• BARTScore: Evaluating Generated Text as Text Generation
• BookSum: A Collection of Datasets for Long-form Narrative Summarization
• QuestEval: Summarization Asks for Fact-based Evaluation
• BLEURT: Learning Robust Metrics for Text Generation
• Evaluating the Factual Consistency of Abstractive Text Summarization
• BERTScore: Evaluating Text Generation with BERT
• FEVER: a large-scale dataset for Fact Extraction and VERification
• TRUE: Re-evaluating Factual Consistency Evaluation

Cited by

• Uncovering and Mitigating Transient Blindness in Multimodal Model Editing

Related

• FactAlign: Long-form Factuality Alignment of Large Language Models
• AlignScore: Evaluating Factual Consistency with a Unified Alignment Function
• Fake Alignment: Are LLMs Really Aligned Well?
• Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
• Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs