On Positional Bias of Faithfulness for Long-form Summarization

2024/10/31 by David Wan, Wan, David, Jesse Vig +5 · 10 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.23609

openalex publication_date 2024/10/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs. We investigate the presence of this bias in long-form summarization, its impact on faithfulness, and various techniques to mitigate this bias. To consistently evaluate faithfulness, we first compile a benchmark of eight human-annotated long-form summarization datasets and perform a meta-evaluation of faithfulness metrics. We show that LLM-based faithfulness metrics, though effective with full-context inputs, remain sensitive to document order, indicating positional bias. Analyzing LLM-generated summaries across six datasets, we find a "U-shaped" trend in faithfulness, where LLMs faithfully summarize the beginning and end of documents but neglect middle content. Perturbing document order similarly reveals models are less faithful when important documents are placed in the middle of the input. We find that this behavior is partly due to shifting focus with context length: as context increases, summaries become less faithful, but beyond a certain length, faithfulness improves as the model focuses on the end. Finally, we experiment with different generation techniques to reduce positional bias and find that prompting techniques effectively direct model attention to specific positions, whereas more sophisticated approaches offer limited improvements. Our data and code are available in https://github.com/meetdavidwan/longformfact.

Cited by

• Enhancing Long Document Long Form Summarisation with Self-Planning
• The Effect of Document Summarization on LLM-Based Relevance Judgments
• Learning from Self Critique and Refinement for Faithful LLM Summarization
• Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts
• Quantifying Cognitive Bias Induction in LLM-Generated Content
• GenerationPrograms: Fine-grained Attribution with Executable Programs
• ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs
• Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge
• Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarization
• QA-prompting: Improving Summarization with Large Language Models using Question-Answering

Related

• On Context Utilization in Summarization with Large Language Models
• LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
• Positional Biases Shift as Inputs Approach Context Window Limits
• Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs
• Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias