2021/06/09 by Tamali Banerjee, Banerjee, Tamali, Rudra Murthy +4
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2106.04995
openalex publication_date 2021/06/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent advances in Unsupervised Neural Machine Translation (UNMT) have\nminimized the gap between supervised and unsupervised machine translation\nperformance for closely related language pairs. However, the situation is very\ndifferent for distant language pairs. Lack of lexical overlap and low syntactic\nsimilarities such as between English and Indo-Aryan languages leads to poor\ntranslation quality in existing UNMT systems. In this paper, we show that\ninitializing the embedding layer of UNMT models with cross-lingual embeddings\nshows significant improvements in BLEU score over existing approaches with\nembeddings randomly initialized. Further, static embeddings (freezing the\nembedding layer weights) lead to better gains compared to updating the\nembedding layer weights during training (non-static). We experimented using\nMasked Sequence to Sequence (MASS) and Denoising Autoencoder (DAE) UNMT\napproaches for three distant language pairs. The proposed cross-lingual\nembedding initialization yields BLEU score improvement of as much as ten times\nover the baseline for English-Hindi, English-Bengali, and English-Gujarati. Our\nanalysis shows the importance of cross-lingual embedding, comparisons between\napproaches, and the scope of improvements in these systems.\n