ARC: Argument Representation and Coverage Analysis for Zero-Shot Long Document Summarization with Instruction Following LLMs

2025/05/29 by Mohamed Elaraby, Elaraby, Mohamed, Diane Litman +1
Computer Science · Social Sciences · #Artificial Intelligence in Law #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.23654

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Argument Representation Coverage (ARC), a bottom-up evaluation framework that assesses how well summaries preserve salient arguments, a crucial issue in summarizing high-stakes domains such as law. ARC provides an interpretable lens by distinguishing between different information types to be covered and by separating omissions from factual errors. Using ARC, we evaluate summaries from eight open-weight large language models in two domains where argument roles are central: long legal opinions and scientific articles. Our results show that while these models capture some salient roles, they frequently omit critical information, particularly when arguments are sparsely distributed across the input. Moreover, ARC uncovers systematic patterns, showing how context window positional bias and role-specific preferences shape argument coverage, and provides actionable guidance for developing more complete and reliable summarization strategies.

Citations

• Behavioral Analysis of Information Salience in Large Language Models
• On Positional Bias of Faithfulness for Long-form Summarization
• UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs
• The Llama 3 Herd of Models
• FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
• MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents
• FABLES: Evaluating faithfulness and content selection in book-length summarization
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
• YaRN: Efficient Context Window Extension of Large Language Models
• Lost in the Middle: How Language Models Use Long Contexts
• Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking
• On Learning to Summarize with Large Language Models as References
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Benchmarking Large Language Models for News Summarization
• LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
• ArgLegalSumm: Improving Abstractive Summarization of Legal Documents with Argument Mining
• SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
• ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining
• Big Bird: Transformers for Longer Sequences
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• HuggingFace's Transformers: State-of-the-art Natural Language Processing

Related

• ArgLegalSumm: Improving Abstractive Summarization of Legal Documents with Argument Mining
• Towards Argument-Aware Abstractive Summarization of Long Legal Opinions with Summary Reranking
• StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMs
• Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing
• STRONG -- Structure Controllable Legal Opinion Summary Generation