AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

2024/01/12 by Li Lucy, Suchin Gururangan, Lucy, Li +12 · 3 voices · 6 citations
Computer Science · Psychology · Social Sciences · #Computer science #Data curation #Data science #Domain (mathematical analysis) #Filter (signal processing) #Identification (biology) #Information retrieval #Natural Language Processing Techniques #Natural language processing #Psychology #Quality (philosophy) #Social media #Topic Modeling #Web page #Wikis in Education and Collaboration #World Wide Web

paper · pdf · doi:10.48550/arxiv.2401.06408

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/01/12 · openalex created_date 2024/01/16 · openalex updated_date 2026/07/28

Abstract

Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage are under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten "quality" and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.

Cited by

• so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs
• Characterizing Narrative Content in Web-scale LLM Pretraining Data
• Analyzing values about gendered language reform in LLMs' revisions
• The Case for Repeatable, Open, and Expert-Grounded Hallucination Benchmarks in Large Language Models
• Enhancing LLMs via High-Knowledge Data Selection
• Beyond Text: Characterizing Domain Expert Needs in Document Research

Discussions

• Cool work here from a team well known to DHers, led by Lucy Li, and including @laurenfklein.bsky.social and @dbamman.bsky.social. Using self-descriptions of 10.3 million website creators, they show t [bsky, 37 points, 2 comments]
• A thread on my latest work! 🧵 We investigate how ten “quality” and English langID filters, drawn from prior lit on LLM pretraining data curation pipelines, affect webpages linked to self-descriptions [bsky, 19 points, 1 comments]
• And there can be all kinds of biases in those datasets, driven not just by data collection but various processing that the datasets go through. For example, many datasets are filtered for "quality" by [bsky, 4 points, 1 comments]

Related

• Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
• Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
• Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation