On Extractive and Abstractive Neural Document Summarization with Transformer Language Models

2019/09/07 by Sandeep Subramanian, Subramanian, Sandeep, Raymond Li +5 · 1 voice · 11 citations
Computer Science · Engineering · #Advanced Text Analysis Techniques #Artificial intelligence #Artificial neural network #Automatic summarization #Computer science #Engineering #Language model #Natural Language Processing Techniques #Natural language processing #Topic Modeling #Transformer #cs.CL

paper · pdf · doi:10.48550/arxiv.1909.03186

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2019/09/07 · arxiv created 2020/04/28 · arxiv updated 2020/04/29 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28

Abstract

We present a method to produce abstractive summaries of long documents that exceed several thousand words via neural abstractive summarization. We perform a simple extractive step before generating a summary, which is then used to condition the transformer language model on relevant information before being tasked with generating a summary. We show that this extractive step significantly improves summarization results. We also show that this approach produces more abstractive summaries compared to prior work that employs a copy mechanism while still achieving higher rouge scores. Note: The abstract above was not written by the authors, it was generated by one of the models presented in this paper.

Citations

• On the difficulty of training Recurrent Neural Networks
• Long Short-Term Memory
• Neural Machine Translation by Jointly Learning to Align and Translate
• Incorporating Copying Mechanism in Sequence-to-Sequence Learning
• WaveNet: A Generative Model for Raw Audio
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• Convolutional Sequence to Sequence Learning
• Mixed Precision Training
• Controlling Decoding for More Abstractive Summaries with Copy-Based Networks
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Neural Summarization by Extracting Sentences and Words

Cited by

• Long document summarization using page specific target text alignment and distilling page importance
• Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling
• Compare: A Framework for Scientific Comparisons
• A Unifying Scheme for Extractive Content Selection Tasks
• An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniques
• HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
• Mobile Application Review Summarization using Chain of Density Prompting
• Neural Language Generation: Formulation, Methods, and Evaluation
• Learning to Summarize Long Texts with Memory Compression and Transfer
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving

Discussions

• On Extractive and Abstractive Summarization with Transformer Language Models [hn, 106 points, 12 comments]

Related

• Repurposing Decoder-Transformer Language Models for Abstractive Summarization
• Combination of abstractive and extractive approaches for summarization of long scientific texts
• A Survey on Neural Abstractive Summarization Methods and Factual Consistency of Summarization
• Controlling the Amount of Verbatim Copying in Abstractive Summarization
• A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents