2019/09/16 by Peter West, West, Peter, Ari Holtzman +5 · 3 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1909.07405
openalex publication_date 2019/09/16 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
The principle of the Information Bottleneck (Tishby et al. 1999) is to\nproduce a summary of information X optimized to predict some other relevant\ninformation Y. In this paper, we propose a novel approach to unsupervised\nsentence summarization by mapping the Information Bottleneck principle to a\nconditional language modelling objective: given a sentence, our approach seeks\na compressed sentence that can best predict the next sentence. Our iterative\nalgorithm under the Information Bottleneck objective searches gradually shorter\nsubsequences of the given sentence while maximizing the probability of the next\nsentence conditioned on the summary. Using only pretrained language models with\nno direct supervision, our approach can efficiently perform extractive sentence\nsummarization over a large corpus.\n Building on our unsupervised extractive summarization (BottleSumEx), we then\npresent a new approach to self-supervised abstractive summarization\n(BottleSumSelf), where a transformer-based language model is trained on the\noutput summaries of our unsupervised method. Empirical results demonstrate that\nour extractive method outperforms other unsupervised models on multiple\nautomatic metrics. In addition, we find that our self-supervised abstractive\nmodel outperforms unsupervised baselines (including our own) by human\nevaluation along multiple attributes.\n