"You are grounded!": Latent Name Artifacts in Pre-trained Language\n Models

2020/04/06 by Vered Shwartz, Shwartz, Vered, Rachel Rudinger +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2004.03012

openalex publication_date 2020/04/06 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

Pre-trained language models (LMs) may perpetuate biases originating in their\ntraining corpus to downstream models. We focus on artifacts associated with the\nrepresentation of given names (e.g., Donald), which, depending on the corpus,\nmay be associated with specific entities, as indicated by next token prediction\n(e.g., Trump). While helpful in some contexts, grounding happens also in\nunder-specified or inappropriate contexts. For example, endings generated for\n`Donald is a' substantially differ from those of other names, and often have\nmore-than-average negative sentiment. We demonstrate the potential effect on\ndownstream tasks with reading comprehension probes where name perturbation\nchanges the model answers. As a silver lining, our experiments suggest that\nadditional pre-training on different corpora may mitigate this bias.\n

Cited by

• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

Related

• Examining the Causal Effect of First Names on Language Models: The Case of Social Commonsense Reasoning
• On the Robustness of Reading Comprehension Models to Entity Renaming
• Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation
• Stubborn Lexical Bias in Data and Models
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning