2021/09/14 by Marc Tanti, Lonneke van der Plas, Tanti, Marc +5 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling #cs.CL #cs.NE
paper · pdf · doi:10.48550/arxiv.2109.06935
14 pages, 6 figures, 5 tables, submitted in BlackBoxNLP 2021 (https://aclanthology.org/2021.blackboxnlp-1.15/)
openalex publication_date 2021/09/14 · arxiv created 2021/12/26 · arxiv updated 2021/12/28 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Recent work has shown evidence that the knowledge acquired by multilingual BERT (mBERT) has two components: a language-specific and a language-neutral one. This paper analyses the relationship between them, in the context of fine-tuning on two tasks -- POS tagging and natural language inference -- which require the model to bring to bear different degrees of language-specific knowledge. Visualisations reveal that mBERT loses the ability to cluster representations by language after fine-tuning, a result that is supported by evidence from language identification experiments. However, further experiments on 'unlearning' language-specific representations using gradient reversal and iterative adversarial learning are shown not to add further improvement to the language-independent component over and above the effect of fine-tuning. The results presented here suggest that the process of fine-tuning causes a reorganisation of the model's limited representational capacity, enhancing language-independent representations at the expense of language-specific ones.