vix.ing · top · new · best · stats · spec

Attention Is All You Need

The Transformer, an attention-only architecture, beat prior best translation results while training far faster than RNN or CNN models.

2017/06/12 by Ashish Vaswani, Vaswani, Ashish, Noam Shazeer +13 · 85 voices · 6522 citations
#cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1706.03762

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Summary

The paper introduces the Transformer, a sequence-to-sequence architecture built entirely out of attention mechanisms, with no recurrence or convolution at all. On two machine translation benchmarks it beat the previous best published models (including ensembles) while training in a fraction of the time, and a version of it also did well on English constituency parsing.

machine-generated · claude-sonnet-5

Outline

machine-generated · claude-sonnet-5

Claims

machine-generated · claude-sonnet-5

Key figure

Figure 1 — A diagram of the full Transformer architecture: the encoder stack on the left and the decoder stack on the right, each made of repeated blocks of multi-head attention and feed-forward layers.

machine-generated · claude-sonnet-5

Glossary

Attention mechanism
A way for a model to weigh and combine information from different parts of its input based on relevance, rather than processing it strictly in order.
Self-attention
Attention applied within a single sequence, letting each position look at every other position in that same sequence to build its representation.
Multi-head attention
Running several attention computations in parallel on different learned projections of the input, then combining the results, so the model can attend to different kinds of relationships at once.
Encoder-decoder
A model structure where one component (encoder) turns an input sequence into an internal representation, and another (decoder) generates the output sequence from that representation.
BLEU score
An automated metric that scores machine translation output by comparing overlapping word sequences against human reference translations; higher is better.
Positional encoding
A signal added to each token's embedding so the model can tell tokens apart by their position in the sequence, since attention alone has no built-in sense of order.
Residual connection / layer normalization
A shortcut that adds a layer's input to its output (helping gradients flow through deep networks), followed by rescaling the result to stabilize training.
Byte-pair / word-piece encoding
A way of splitting text into subword units so a fixed, moderate-size vocabulary can still represent rare and unseen words.
Beam search
A decoding strategy that keeps several of the most likely partial output sequences at each step instead of only the single best one.
Label smoothing
A training trick that discourages the model from becoming overconfident in its predictions, which lowers perplexity but improves BLEU and accuracy.

machine-generated · claude-sonnet-5

Audience

Machine learning practitioners and researchers working on sequence modeling, NLP, or machine translation who want to understand the origin of the Transformer architecture underlying most modern language models.

prerequisites: Basic neural network training concepts, Familiarity with recurrent and convolutional sequence-to-sequence models, Linear algebra (matrix multiplication, vector spaces), Some prior exposure to attention mechanisms in encoder-decoder models

machine-generated · claude-sonnet-5

Open questions

machine-generated · claude-sonnet-5

Supplementary links

machine-generated · claude-sonnet-5

Cited by

Discussions

Related