Transformer Quality in Linear Time

2022/02/21 by Weizhe Hua, Hua, Weizhe, Zihang Dai +5 · 11 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2202.10447

openalex publication_date 2022/02/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9× on Wiki-40B and 12.1× on PG-19 for auto-regressive language modeling, and 4.8× on C4 for masked language modeling.

Cited by

• KOSS: Kalman-Optimal Selective State Spaces for Long-Term Sequence Modeling
• Deep Learning-Based Joint Uplink-Downlink CSI Acquisition for Next-Generation Upper Mid-Band Systems
• TNT: Improving Chunkwise Training for Test-Time Memorization
• GroupKAN: Efficient Kolmogorov-Arnold Networks via Grouped Spline Modeling
• FlashEVA: Accelerating LLM inference via Efficient Attention
• Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency
• Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• FLASepformer: Efficient Speech Separation with Gated Focused Linear Attention Transformer
• Efficient Attention Mechanisms for Large Language Models: A Survey

Related

• Do Transformers Need Deep Long-Range Memory
• Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
• Simplifying Transformer Blocks
• Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers
• Long Range Arena: A Benchmark for Efficient Transformers