Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning

2024/05/13 by Jisu Kim, Kim, Jisu, Ju-Hwan Lee +1 · 10 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.07490

openalex publication_date 2024/05/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The rapid advancement of Large Language Models (LLMs) has improved text understanding and generation but poses challenges in computational resources. This study proposes a curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Experiments with Mistral-7B (Jiang et al., 2023) and Gemma-7B (Team et al., 2024) models demonstrate that curriculum learning slightly improves performance compared to traditional random data shuffling. Notably, we observed that sorting data based on our proposed attention criteria generally led to better performance. This approach offers a sustainable method to enhance LLM performance without increasing model size or dataset volume, addressing scalability challenges in LLM training.

Cited by

• From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
• How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• General Intelligence-based Fragmentation (GIF): A framework for peak-labeled spectra simulation
• Revisiting Multilingual Data Mixtures in Language Model Pretraining
• What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity
• Data Efficacy for Language Model Training
• Complexity-aware fine-tuning
• Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data
• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning

Related

• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• Influence-driven Curriculum Learning for Pre-training on Limited Data
• Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum
• Data Efficacy for Language Model Training
• Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining