Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models

2024/10/21 by Divyanshu Aggarwal, Ashutosh Sathe, Aggarwal, Divyanshu +3 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.16168

openalex publication_date 2024/10/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) demonstrate exceptional capabilities in a multitude of NLP tasks. However, the efficacy of such models to languages other than English is often limited. Prior works have shown that encoder-only models such as BERT or XLM-RoBERTa show impressive cross lingual transfer of their capabilities from English to other languages. In this work, we propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. We show that LLMs pretrained with active forgetting are highly effective when adapting to new and unseen languages. Through extensive experimentation, we find that LLMs pretrained with active forgetting are able to learn better multilingual representations which translates to better performance in many downstream tasks.

Cited by

• SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation
• Continually Adding New Languages to Multilingual Language Models

Related

• Improving Language Plasticity via Pretraining with Active Forgetting
• Efficiently Adapting Pretrained Language Models To New Languages
• Mitigating Catastrophic Forgetting in Language Transfer via Model Merging
• How Do Multilingual Encoders Learn Cross-lingual Representation?
• On the ability of monolingual models to learn language-agnostic representations