Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size

2020/08/16 by Davis Yoshida, Allyson Ettinger, Yoshida, Davis +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Music and Audio Processing #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2008.07027

openalex publication_date 2020/08/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years. While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures. We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time. An additional benefit is that our method removes the fixed context size constraint that most transformer models have, allowing for more flexible use. When applied to the GPT-2 language model, we find that our method attains better perplexity than an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.

Citations

• Training Deep Nets with Sublinear Memory Cost
• Pointer Sentinel Mixture Models
• Pay Less Attention with Lightweight and Dynamic Convolutions
• LLM-Supported Manufacturing Mapping Generation
• Reformer: The Efficient Transformer
• What the [MASK]? Making Sense of Language-Specific BERT Models
• Longformer: The Long-Document Transformer
• Synthesizer: Rethinking Self-Attention in Transformer Models

Related

• Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers
• Memory-efficient Stochastic methods for Memory-based Transformers
• Simple Recurrence Improves Masked Language Models
• Primer: Searching for Efficient Transformers for Language Modeling
• Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models