ETC: Encoding Long and Structured Inputs in Transformers

2020/04/17 by Joshua Ainslie, Ainslie, Joshua, Santiago Ontañón +19 · 1 voice · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2004.08483

Accepted at EMNLP 2020

openalex publication_date 2020/04/17 · arxiv published 2020/04/17 · arxiv created 2020/10/27 · arxiv updated 2020/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

Citations

• Practical and Optimal LSH for Angular Distance
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient\n Sequence Modeling
• Generating Wikipedia by Summarizing Long Sequences
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• A BERT Baseline for the Natural Questions
• Generating Long Sequences with Sparse Transformers
• HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Generating Logical Forms from Graph Representations of Text and Entities
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Adaptively Sparse Transformers
• Compressive Transformers for Long-Range Sequence Modelling
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Longformer: The Long-Document Transformer
• RikiNet: Reading Wikipedia Pages for Natural Question Answering

Cited by

• DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration

Discussions

• arxiv.org/pdf/2004.08483 ETC, which is similar to Longformer. arxiv.org/pdf/2006.03274 GMAT, also similar to Longformer. arxiv.org/pdf/2202.10890 Hierarchical Perceiver. [bsky, 0 points, 0 comments]

Related

• Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
• Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers
• Introduction to Transformers: an NLP Perspective
• A Length-Extrapolatable Transformer
• Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks