Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning

2021/08/30 by Ran Tian, Tian, Ran, Joshua Maynez +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2108.13032

openalex publication_date 2021/08/30 · openalex created_date 2021/09/13 · openalex updated_date 2026/07/28

Abstract

The highly popular Transformer architecture, based on self-attention, is the foundation of large pretrained models such as BERT, that have become an enduring paradigm in NLP. While powerful, the computational resources and time required to pretrain such models can be prohibitive. In this work, we present an alternative self-attention architecture, Shatter, that more efficiently encodes sequence information by softly partitioning the space of relative positions and applying different value matrices to different parts of the sequence. This mechanism further allows us to simplify the multi-headed attention in Transformer to single-headed. We conduct extensive experiments showing that Shatter achieves better performance than BERT, with pretraining being faster per step (15% on TPU), converging in fewer steps, and offering considerable memory savings (>50%). Put together, Shatter can be pretrained on 8 V100 GPUs in 7 days, and match the performance of BERT_Base -- making the cost of pretraining much more affordable.

Citations

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Are Sixteen Heads Really Better than One?
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Reformer: The Efficient Transformer
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Longformer: The Long-Document Transformer
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• The MultiBERTs: BERT Reproductions for Robustness Analysis

Related

• Fast Transformer Decoding: One Write-Head is All You Need
• Conv-Linformer: Boosting Linformer's Performance with Convolution in Small-Scale Settings
• Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers
• Attention Is Not All You Need Anymore
• ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs