CiteEval: Principle-Driven Citation Evaluation for Source Attribution

2025/06/02 by Yumo Xu, Peng Qi, Xu, Yumo +17 · 1 voice · 8 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Benchmark (surveying) #Biomedical Text Mining and Ontologies #Citation #Construct (python library) #Inference #Proxy (statistics) #Quality (philosophy) #Scalability #Suite #cs.AI #cs.CL #cs.IR

paper · pdf · doi:10.48550/arxiv.2506.01829

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/02 · arxiv published 2025/06/02 · arxiv updated 2025/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Citation quality is crucial in information-seeking systems, directly influencing trust and the effectiveness of information access. Current evaluation frameworks, both human and automatic, mainly rely on Natural Language Inference (NLI) to assess binary or ternary supportiveness from cited sources, which we argue is a suboptimal proxy for citation evaluation. In this work we introduce CiteEval, a citation evaluation framework driven by principles focusing on fine-grained citation assessment within a broad context, encompassing not only the cited sources but the full retrieval context, user query, and generated text. Guided by the proposed framework, we construct CiteBench, a multi-domain benchmark with high-quality human annotations on citation quality. To enable efficient evaluation, we further develop CiteEval-Auto, a suite of model-based metrics that exhibit strong correlation with human judgments. Experiments across diverse systems demonstrate CiteEval-Auto's superior ability to capture the multifaceted nature of citations compared to existing metrics, offering a principled and scalable approach to evaluate and improve model-generated citations.

Citations

• MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
• Personalization of Large Language Models: A Survey
• LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Llama 3 Herd of Models
• RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
• Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics
• Learning to Plan and Generate Text with Citations
• FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability
• Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks
• ExpertQA: Expert-Curated Questions and Attributed Answers
• Enabling Large Language Models to Generate Text with Citations
• Evaluating and Modeling Attribution for Cross-Lingual Question Answering
• Evaluating Verifiability in Generative Search Engines
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• ASQA: Factoid Questions Meet Long-Form Answers
• Teaching language models to support answers with verified quotes
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• ELI5: Long Form Question Answering
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset

Cited by

• Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore
• Epistemic Substitution: How Grokipedia's AI-Generated Encyclopedia Restructures Authority
• Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
• Exposing Citation Vulnerabilities in Generative Engines
• Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement
• Architecting Trust in Artificial Epistemic Agents
• Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact
• From Citation Selection to Citation Absorption: A Measurement Framework for Generative Engine Optimization Across AI Search Platforms

Discussions

• b) as builders, we evaluate the technology soberly and help users navigate these risks in product design. Want to learn more? Checkout Our paper: arxiv.org/pdf/2506.01829 Open-source code: github.com/ [bsky, 0 points, 1 comments]

Related

• CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validation
• ALiiCE: Evaluating Positional Fine-grained Citation Generation
• Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics
• CiteBench: A benchmark for Scientific Citation Text Generation
• What Should I Cite? A RAG Benchmark for Academic Citation Prediction