Efficient Attentions for Long Document Summarization

2021/04/05 by Luyang Huang, Huang, Luyang, Shuyang Cao +7 · 36 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2104.02112

openalex publication_date 2021/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The quadratic computational and memory complexities of large Transformers have limited their scalability for long document summarization. In this paper, we propose Hepos, a novel efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. We further conduct a systematic study of existing efficient self-attentions. Combined with Hepos, we are able to process ten times more tokens than existing models that use full attentions. For evaluation, we present a new dataset, GovReport, with significantly longer documents and summaries. Results show that our models produce significantly higher ROUGE scores than competitive comparisons, including new state-of-the-art results on PubMed. Human evaluation also shows that our models generate more informative summaries with fewer unfaithful errors.

Citations

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Reformer: The Efficient Transformer

Cited by

• AgentWatcher: A Rule-based Prompt Injection Monitor
• Accelerate Speculative Decoding with Sparse Computation in Verification
• Enhancing Long Document Long Form Summarisation with Self-Planning
• Compass: Co-Exploration of Mapping and Hardware for Heterogeneous Multi-Chiplet Accelerators Targeting LLM Inference Service Workloads
• What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models
• SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
• ScaleFormer: Span Representation Cumulation for Long-Context Transformer
• SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification
• Stress Testing Factual Consistency Metrics for Long-Document Summarization
• Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
• Decomposition-Enhanced Training for Post-Hoc Attributions In Language Models
• Citation Failure: Definition, Analysis and Efficient Mitigation
• Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
• AcademicEval: Live Long-Context LLM Benchmark
• LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA
• Taming the Fragility of KV Cache Eviction in LLM Inference
• Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
• Revisiting Long-context Modeling from Context Denoising Perspective
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture
• SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
• Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
• Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization
• Mamba Modulation: On the Length Generalization of Mamba
• SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
• HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
• Long document summarization using page specific target text alignment and distilling page importance
• Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
• From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
• ScaleDoc: Scaling LLM-based Predicates over Large Document Collections
• HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking
• EvolKV: Evolutionary KV Cache Compression for LLM Inference
• HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
• PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
• SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
• SCOPE: A Generative Approach for LLM Prompt Compression

Related

• Long Document Summarization with Top-down and Bottom-up Inference
• Sparsity and Sentence Structure in Encoder-Decoder Attention of\n Summarization Systems
• HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization
• A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
• Long-Span Summarization via Local Attention and Content Selection