Blockwise Self-Attention for Long Document Understanding

2019/11/07 by Jiezhong Qiu, Qiu, Jiezhong, Hao Ma +9 · 6 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1911.02972

Accepted at Findings of EMNLP'20 and SustaiNLP 2020 at EMNLP'20, 12 pages

arxiv created 2020/11/01 · arxiv updated 2020/11/03

Abstract

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Cited by

• PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
• DARE: An Irregularity-Tolerant Matrix Processing Unit with a Densifying ISA and Filtered Runahead Execution
• Optimizing Mixture of Block Attention
• Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
• When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
• Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light

Related

• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Revealing the Dark Secrets of BERT
• SesameBERT: Attention for Anywhere
• TrimBERT: Tailoring BERT for Trade-offs
• LittleBird: Efficient Faster & Longer Transformer for Question Answering