Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
2026/01/15 by Cameron Tice, Puria Radmard, Samuel Ratnam +3 · 12 voices · 2 citations
#cs.CL #cs.AI #cs.LG
paper · pdf
Abstract
Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descriptions of AI behaviour are predominantly negative, LLMs may internalise corresponding behavioural priors, giving rise to self-fulfilling misalignment. This paper provides the first controlled study of this hypothesis by pretraining 6.9B-parameter LLMs with varying amounts of (mis)alignment discourse. We find that discussion of AI contributes to misalignment. Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behaviour. Conversely, upsampling documents about aligned behaviour reduces misalignment scores from 45% to 9%. We consider this evidence of self-fulfilling alignment. These effects are dampened, but persist through post-training. Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training. We recommend practitioners consider pretraining for alignment alongside capabilities. We share our models, data, and evaluations at AlignmentPretraining.ai.
Cited by
Discussions
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment [hn, 81 points, 30 comments]
- AI한테 AI가 인간 배신하고 나쁜 짓하는 이야기를 학습시키니 나쁜 행동을 할 가능성이 높아졌다고 한다. 반대로 AI가 유혹에 빠지지 않고 인간을 위해 행동하는 이야기를 학습시키니 그 반대로 선한 선택을 했다고. Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment arxiv.o [bsky, 3 points, 0 comments]
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment https://arxiv.org/abs/26... [bsky, 2 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment https://arxiv.org/abs/2601.10160 [comments] [26 points] [bsky, 1 points, 0 comments]
- 📰 Alignment pretraining in AI discourse can lead to self-fulfilling misalignment, as it may inadvertently reinforce biases and create unintended consequences. 🔗 https://arxiv.org/abs/2601.10160 #Tec [bsky, 1 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment https://arxiv.org/abs/2601.10160 (https://news.ycombinator.com/item?id=48185938) [bsky, 0 points, 0 comments]
- I was looking for a prior bit of work on this topic and found this instead which looks even more exactly about it. arxiv.org/abs/2601.101... [bsky, 0 points, 0 comments]
- Exploring the implications of alignment pretraining in AI discourse reveals a cycle of self-fulfilling misalignment. As we navigate these complex dynamics, understanding and addressing these challenge [bsky, 0 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment https:// arxiv.org/abs/2601.10160 # ai # arxiv [mastodon, 0 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment https://arxiv.org/abs/2601.10160 (https://news.ycombinator.com/item?id=48185938) [bsky, 0 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment [bsky, 0 points, 0 comments]
- Alignment pretraining: AI discourse creates self-fulfilling (mis)alignment https://arxiv.org/abs/2601.10160 [bsky, 0 points, 0 comments]
Related