Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

2022/03/19 by Ryokan Ri, Ri, Ryokan, Yoshimasa Tsuruoka +1 · 4 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2203.10326

openalex publication_date 2022/03/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language. A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language. Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.

Cited by

• Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
• Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
• Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
• Transfer of Structural Knowledge from Synthetic Languages

Related

• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
• Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
• Pre-Training a Language Model Without Human Language
• Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation
• Linguistic Knowledge and Transferability of Contextual Representations