Attention Strategies for Multi-Source Sequence-to-Sequence Learning

2017/04/21 by Jindřich Libovický, Libovický, Jindřich, Jindřich Helcl +1 · 1 citation
Computer Science · #68T50 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Neural and Evolutionary Computing (cs.NE) #acm:68T50 #cs.CL #cs.NE #msc:68T50

paper · pdf · doi:10.48550/arxiv.1704.06567

7 pages; Accepted to ACL 2017

arxiv created 2017/04/21 · arxiv updated 2017/04/24

Abstract

Modeling attention in neural multi-source sequence-to-sequence learning remains a relatively unexplored area, despite its usefulness in tasks that incorporate multiple source languages or modalities. We propose two novel approaches to combine the outputs of attention mechanisms over each source sequence, flat and hierarchical. We compare the proposed methods with existing techniques and present results of systematic evaluation of those methods on the WMT16 Multimodal Translation and Automatic Post-editing tasks. We show that the proposed methods achieve competitive results on both tasks.

Cited by

• Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decoding

Related

• Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model
• Attention Forcing for Machine Translation
• Hierarchical Learning for Generation with Long Source Sequences
• Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
• Monotonic Multihead Attention