2019/06/28 by Denis Emelin, Ivan Titov, Emelin, Denis +3
Computer Science · Engineering · #Artificial intelligence #Artificial neural network #Bottleneck #Computation and Language (cs.CL) #Computer science #Embedding #Encoder #Engineering #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine translation #Natural Language Processing Techniques #Natural language processing #Speech recognition #Text Readability and Simplification #Topic Modeling #Transformer #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.1906.12284
published in arXiv (Cornell University) (Cornell University) · Accepted submission to WMT 2019
arxiv created 2019/06/28 · openalex publication_date 2019/06/28 · arxiv updated 2019/07/01 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context. Lexical features are fed into the first layer and propagated through a deep network of hidden layers. We argue that the need to represent and propagate lexical features in each layer limits the model's capacity for learning and representing other information relevant to the task. To alleviate this bottleneck, we introduce gated shortcut connections between the embedding layer and each subsequent layer within the encoder and decoder. This enables the model to access relevant lexical content dynamically, without expending limited resources on storing it within intermediate states. We show that the proposed modification yields consistent improvements over a baseline transformer on standard WMT translation tasks in 5 translation directions (0.9 BLEU on average) and reduces the amount of lexical information passed along the hidden layers. We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.