Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

2026/01/01 by Atsuki Yamaguchi, Maggie Mi, Nikolaos Aletras · 1 citation
Arts and Humanities · Computer Science · Social Sciences · #EFL/ESL Teaching and Learning #Intelligent Tutoring Systems and Adaptive Learning #Writing and Handwriting Education #cs.CL

paper · pdf · doi:10.18653/v1/2026.acl-short.27

Accepted to ACL 2026 Main Conference

openalex publication_date 2026/01/01 · arxiv created 2026/04/15 · openalex created_date 2026/07/02 · openalex updated_date 2026/07/29 · arxiv updated 2026/07/30

Abstract

Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.

Citations

• Linguistic Interpretability of Transformer-based Language Models: a systematic review
• Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Instruction Pre-Training: Language Models are Supervised Multitask Learners
• Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence
• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
• Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training
• Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?
• Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT
• How does the task complexity of masked pretraining objectives affect downstream performance?
• Language Model Behavior: A Comprehensive Survey
• Language Model Behavior: A Comprehensive Survey
• Deep neural networks and humans both benefit from compositional language structure
• Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization
• Efficient Training of Language Models to Fill in the Middle
• Large Language Models are Zero-Shot Reasoners
• Improving In-Context Few-Shot Learning via Self-Supervised Training
• Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
• How does the pre-training objective affect what large language models learn about linguistic properties?
• Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
• Frustratingly Simple Pretraining Alternatives to Masked Language Modeling
• Finetuned Language Models Are Zero-Shot Learners
• On the Dangers of Stochastic Parrots
• Probing Classifiers: Promises, Shortcomings, and Advances
• Syntax-Enhanced Pre-trained Model
• On Losses for Modern Language Models
• Language Models are Few-Shot Learners
• A Primer in BERTology: What we know about how BERT works
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• HellaSwag: Can a Machine Really Finish Your Sentence?
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Crowdsourcing Multiple Choice Science Questions
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• The LAMBADA dataset: Word prediction requiring a broad discourse context
• Learning and development in neural networks: the importance of starting small
• A NEW MEASURE OF RANK CORRELATION
• Adapting Large Language Models to Domains via Reading Comprehension
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

Related

• Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap
• Thinking Augmented Pre-training
• LERT: A Linguistically-motivated Pre-trained Language Model
• Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models
• Reinforcement Pre-Training