Transfer of Structural Knowledge from Synthetic Languages

2025/05/21 by Mikhail Budnikov, Ivan P. Yamshchikov, Budnikov, Mikhail +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.15769

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

Citations

• SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Injecting structural hints: Using language models to study inductive biases in language learning
• Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
• Does Pretraining for Summarization Require Knowledge Transfer?
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
• Pre-Trained Models: Past, Present and Future
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
• Pretrained Transformers as Universal Computation Engines
• LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning
• Measuring Massive Multitask Language Understanding
• Universal linguistic inductive biases via meta-learning
• On the Cross-lingual Transferability of Monolingual Representations
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Long Short-Term Memory

Related

• Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
• Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models
• BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer
• Using natural language prompts for machine translation
• Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning