2020/10/24 by Alexander R. Fabbri, Fabbri, Alexander R., Simeng Han +13 · 1 citation
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2010.12836
Models pretrained with self-supervised objectives on large text corpora\nachieve state-of-the-art performance on English text summarization tasks.\nHowever, these models are typically fine-tuned on hundreds of thousands of data\npoints, an infeasible requirement when applying summarization to new, niche\ndomains. In this work, we introduce a novel and generalizable method, called\nWikiTransfer, for fine-tuning pretrained models for summarization in an\nunsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained\nmodels on pseudo-summaries, produced from generic Wikipedia data, which contain\ncharacteristics of the target dataset, such as the length and level of\nabstraction of the desired summaries. WikiTransfer models achieve\nstate-of-the-art, zero-shot abstractive summarization performance on the\nCNN-DailyMail dataset and demonstrate the effectiveness of our approach on\nthree additional diverse datasets. These models are more robust to noisy data\nand also achieve better or comparable few-shot performance using 10 and 100\ntraining examples when compared to few-shot transfer from other summarization\ndatasets. To further boost performance, we employ data augmentation via\nround-trip translation as well as introduce a regularization term for improved\nfew-shot transfer. To understand the role of dataset aspects in transfer\nperformance and the quality of the resulting output summaries, we further study\nthe effect of the components of our unsupervised fine-tuning data and analyze\nfew-shot performance using both automatic and human evaluation.\n