Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding

2025/07/13 by Feng Qi, Feng, Qi, Yihong Liu +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.6 #I.2.7 #Intelligent Tutoring Systems and Adaptive Learning #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.09758

openalex publication_date 2025/07/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Curriculum learning is a widely adopted training strategy in natural language processing (NLP), where models are exposed to examples organized by increasing difficulty to enhance learning efficiency and performance. However, most existing approaches rely on manually defined difficulty metrics -- such as text length -- which may not accurately reflect the model's own perspective. To overcome this limitation, we present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) themselves. Building on these scores, we explore various training strategies that differ in the ordering of examples for the fine-tuning: from easy-to-hard, hard-to-easy, to mixed sampling. We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks. Experimental results show that our approach leads to faster convergence and improved performance compared to standard random sampling.

Citations

• LangSAMP: Language-Script Aware Multilingual Pretraining
• SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists
• Curriculum Learning for Small Code Language Models
• To Label or Not to Label: Hybrid Active Learning for Neural Machine Translation
• Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal
• Balanced Data Sampling for Language Model Training with Clustering
• Self-Influence Guided Data Reweighting for Language Model Pre-training
• Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum
• Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
• Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data
• AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages
• Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding
• Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification
• NGAME: Negative Mining-aware Mini-batching for Extreme Classification
• Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• How Many Data Points is a Prompt Worth?
• Making Pre-trained Language Models Better Few-shot Learners
• Few-Shot Text Generation with Pattern-Exploiting Training
• Cold-start Active Learning through Self-supervised Language Modeling
• Self-training Improves Pre-training for Natural Language Understanding
• JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot\n Learners
• A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation
• Exploiting Cloze Questions for Few Shot Text Classification and Natural\n Language Inference
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• XNLI: Evaluating Cross-lingual Sentence Representations
• Automated Hate Speech Detection and the Problem of Offensive Language
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation

Related

• Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU
• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• Self-training with Few-shot Rationalization: Teacher Explanations Aid Student in Few-shot NLU
• Influence-driven Curriculum Learning for Pre-training on Limited Data
• Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning