Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning

2020/04/29 by Zihan Liu, Liu, Zihan, Genta Indra Winata +5 · 3 citations
Computer Science · #Topic Modeling #Speech Recognition and Synthesis #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2004.14218

Abstract

Recently, fine-tuning pre-trained language models (e.g., multilingual BERT) to downstream cross-lingual tasks has shown promising results. However, the fine-tuning process inevitably changes the parameters of the pre-trained model and weakens its cross-lingual ability, which leads to sub-optimal performance. To alleviate this problem, we leverage continual learning to preserve the original cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. The experimental result shows that our fine-tuning methods can better preserve the cross-lingual ability of the pre-trained model in a sentence retrieval task. Our methods also achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.

Cited by

• Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
• mT5: A massively multilingual pre-trained text-to-text transformer
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment

Related

• Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models
• Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding
• Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model
• CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
• Consistency Regularization for Cross-Lingual Fine-Tuning