ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems

2019/04/04 by Unanue, Inigo Jauregi, Borzeshi, Ehsan Zare, Esmaili, Nazanin +1
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1904.02461

Abstract

Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in a system that is jointly trained to predict the next word in the translation (categorical value) and its word embedding (continuous value). Such a joint training allows the proposed system to learn the distributional properties represented by the word embeddings, empirically improving the generalization to unseen sentences. Experiments over three translation datasets have showed a consistent improvement over a strong baseline, ranging between 0.91 and 2.54 BLEU points, and also a marked improvement over a state-of-the-art system.

Related

• Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation
• Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
• Regularization techniques for fine-tuning in neural machine translation
• When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?
• Subword Regularization: Improving Neural Network Translation Models with\n Multiple Subword Candidates