Transformer on a Diet

2020/02/14 by Chenguang Wang, Zihao Ye, Wang, Chenguang +7
Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2002.06170

openalex publication_date 2020/02/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.

Citations

• Pointer Sentinel Mixture Models
• Language Models with Transformers
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Related

• A Survey of Techniques for Optimizing Transformer Inference
• Bag of Tricks for Optimizing Transformer Efficiency
• Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference
• DeLighT: Deep and Light-weight Transformer
• Do Transformer Modifications Transfer Across Implementations and Applications?