Sparse is Enough in Scaling Transformers

2021/11/24 by Sebastian Jaszczur, Aakanksha Chowdhery, Jaszczur, Sebastian +11 · 7 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2111.12763

openalex publication_date 2021/11/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach. We address this problem by leveraging sparsity. We study sparse variants for all layers in the Transformer and propose Scaling Transformers, a family of next generation Transformer models that use sparse layers to scale efficiently and perform unbatched decoding much faster than the standard Transformer as we scale up the model size. Surprisingly, the sparse layers are enough to obtain the same perplexity as the standard Transformer with the same number of parameters. We also integrate with prior sparsity approaches to attention and enable fast inference on long sequences even with limited memory. This results in performance competitive to the state-of-the-art on long text summarization.

Cited by

• Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifolds
• Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders
• Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning
• Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
• FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning
• Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
• Spatial-Temporal-Spectral Mamba with Sparse Deformable Token Sequence for Enhanced MODIS Time Series Classification

Related

• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
• Transformer Acceleration with Dynamic Sparse Attention
• Long Range Arena: A Benchmark for Efficient Transformers
• SPION: Layer-Wise Sparse Training of Transformer via Convolutional Flood Filling