2024/10/17 by Yiming Zhang, Javier Rando, Zhang, Yiming +15 · 3 voices · 32 citations
Computer Science · Medicine · Psychology · #Geography #Medicine #Meteorology #Poisoning and overdose treatments #Psychology #Training (meteorology) #cs.AI #cs.CR
paper · pdf · doi:10.48550/arxiv.2410.13722
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Large language models are pre-trained on uncurated text datasets consisting of trillions of tokens scraped from the Web. Prior work has shown that: (1) web-scraped pre-training datasets can be practically poisoned by malicious actors; and (2) adversaries can compromise language models after poisoning fine-tuning datasets. Our work evaluates for the first time whether language models can also be compromised during pre-training, with a focus on the persistence of pre-training attacks after models are fine-tuned as helpful and harmless chatbots (i.e., after SFT and DPO). We pre-train a series of LLMs from scratch to measure the impact of a potential poisoning adversary under four different attack objectives (denial-of-service, belief manipulation, jailbreaking, and prompt stealing), and across a wide range of model sizes (from 600M to 7B). Our main result is that poisoning only 0.1% of a model's pre-training dataset is sufficient for three out of four attacks to measurably persist through post-training. Moreover, simple attacks like denial-of-service persist through post-training with a poisoning rate of only 0.001%.