2021/06/03 by Kaushal Kumar Maurya, Maurya, Kaushal Kumar, Maunendra Sankar Desarkar +5
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2106.01597
openalex publication_date 2021/06/03 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Despite the recent advancement in NLP research, cross-lingual transfer for\nnatural language generation is relatively understudied. In this work, we\ntransfer supervision from high resource language (HRL) to multiple low-resource\nlanguages (LRLs) for natural language generation (NLG). We consider four NLG\ntasks (text summarization, question generation, news headline generation, and\ndistractor generation) and three syntactically diverse languages, i.e.,\nEnglish, Hindi, and Japanese. We propose an unsupervised cross-lingual language\ngeneration framework (called ZmBART) that does not use any parallel or\npseudo-parallel/back-translated data. In this framework, we further pre-train\nmBART sequence-to-sequence denoising auto-encoder model with an auxiliary task\nusing monolingual data of three languages. The objective function of the\nauxiliary task is close to the target tasks which enriches the multi-lingual\nlatent representation of mBART and provides good initialization for target\ntasks. Then, this model is fine-tuned with task-specific supervised English\ndata and directly evaluated with low-resource languages in the Zero-shot\nsetting. To overcome catastrophic forgetting and spurious correlation issues,\nwe applied freezing model component and data argumentation approaches\nrespectively. This simple modeling approach gave us promising results.We\nexperimented with few-shot training (with 1000 supervised data points) which\nboosted the model performance further. We performed several ablations and\ncross-lingual transferability analyses to demonstrate the robustness of ZmBART.\n