Alleviating the Inequality of Attention Heads for Neural Machine Translation

2020/09/21 by Zewei Sun, Shujian Huang, Sun, Zewei +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2009.09672

openalex publication_date 2020/09/21 · openalex created_date 2020/09/25 · openalex updated_date 2026/07/28

Abstract

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.

Related

• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Fixed Encoder Self-Attention Patterns in Transformer-Based Machine\n Translation
• A Dynamic Head Importance Computation Mechanism for Neural Machine Translation
• Hard-Coded Gaussian Attention for Neural Machine Translation
• Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads?