HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization

2021/10/12 by Liu, Ye, Zhang, Jian-Guo, Wan, Yao +3 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2110.06388

Abstract

To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1 while using less memory and fewer parameters.

Cited by

• Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization

Related

• HEGEL: Hypergraph Transformer for Long Document Summarization
• GraphLSS: Integrating Lexical, Structural, and Semantic Features for Long Document Extractive Summarization
• Long-Span Summarization via Local Attention and Content Selection
• Hierarchical Transformers for Multi-Document Summarization
• Heterogeneous Graph Neural Networks for Extractive Document Summarization