2023/11/02 by Megh Thakkar, Thakkar, Megh, Tolga Bolukbasi +9 · 10 citations
Computer Science · Mathematics · #Artificial intelligence #Computer science #Context (archaeology) #Language model #Machine learning #Mathematics #Natural Language Processing Techniques #Natural language processing #Novelty #Point (geometry) #Quality (philosophy) #Relevance (law) #Sample (material) #Stability (learning theory) #Task (project management) #Topic Modeling #Training set
paper · pdf · doi:10.48550/arxiv.2311.00913
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Language Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are treated with equal importance during LM pre-training. However, due to varying levels of relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored in the context of task-specific supervised learning and LM fine-tuning, model-driven reweighting for pre-training data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores as an indicator of sample importance and pre-training. PRESENCE promotes novelty and stability for model pre-training. Through extensive analysis spanning multiple model sizes, datasets, and tasks, we present PRESENCE as an important first step in the research direction of sample reweighting for pre-training language models.