Faster Transformer Decoding: N-gram Masked Self-Attention

2020/01/14 by Ciprian Chelba, Chelba, Ciprian, Mia Xu Chen +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2001.04589

openalex publication_date 2020/01/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Motivated by the fact that most of the information relevant to the prediction of target tokens is drawn from the source sentence S=s₁, …, s_S, we propose truncating the target-side window used for computing self-attention by making an N-gram assumption. Experiments on WMT EnDe and EnFr data sets show that the N-gram masked self-attention model loses very little in BLEU score for N values in the range 4, …, 8, depending on the task.

Citations

• Fast Transformer Decoding: One Write-Head is All You Need

Cited by

• Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Related

• Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attention
• Integral Transformer: Denoising Attention, Not Too Much Not Too Little
• Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
• kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers
• Sharing Attention Weights for Fast Transformer