Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

2024/09/30 by David Grangier, Simin Fan, Grangier, David +5 · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2410.03735

openalex publication_date 2024/09/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain these models is only available in limited amount for most tasks. In this work, we build specialist models from large generalist training sets instead. We propose a novel method, ClusteRed Importance SamPling (CRISP). CRISP clusters the generalist dataset and samples from these clusters based on their frequencies in the smaller specialist dataset. It is scalable, suitable for both pretraining and continued pretraining, and works well in multi-task settings. CRISP performs favorably compared to other methods that adjust the training distribution of the generalist data with guidance from the limited domain-specific data. Our findings demonstrate improvements across different domains in terms of language modeling perplexity and accuracy on multiple-choice question tasks. We also present ablation studies that examine the impact of dataset sizes, clustering configurations, and model sizes.

Cited by

• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
• Closing the Gap Between Text and Speech Understanding in LLMs
• The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
• Pretraining with hierarchical memories: separating long-tail and common knowledge
• LLM on a Budget: Active Knowledge Distillation for Efficient Classification of Large Text Corpora
• Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation
• Language Models Improve When Pretraining Data Matches Target Tasks
• Scaling Laws for Optimal Data Mixtures

Related

• Target-Aware Language Modeling via Granular Data Sampling
• Balanced Data Sampling for Language Model Training with Clustering
• Harnessing Diversity for Important Data Selection in Pretraining Large Language Models
• Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model
• Need a Small Specialized Language Model? Plan Early!