Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models
2025/07/10 by K.S. Liang, Kaiqu Liang, Liang, Kaiqu +10 · 15 voices · 2 citations
Social Sciences · Computer Science · #Misinformation and Its Impacts #Hate Speech and Cyberbullying Detection #Computational and Text Analysis Methods
paper · pdf · doi:10.48550/arxiv.2507.07484
Abstract
Bullshit, as conceptualized by philosopher Harry Frankfurt, refers to statements made without regard to their truth value. While previous work has explored large language model (LLM) hallucination and sycophancy, we propose machine bullshit as an overarching conceptual framework that can allow researchers to characterize the broader phenomenon of emergent loss of truthfulness in LLMs and shed light on its underlying mechanisms. We introduce the Bullshit Index, a novel metric quantifying LLMs' indifference to truth, and propose a complementary taxonomy analyzing four qualitative forms of bullshit: empty rhetoric, paltering, weasel words, and unverified claims. We conduct empirical evaluations on the Marketplace dataset, the Political Neutrality dataset, and our new BullshitEval benchmark (2,400 scenarios spanning 100 AI assistants) explicitly designed to evaluate machine bullshit. Our results demonstrate that model fine-tuning with reinforcement learning from human feedback (RLHF) significantly exacerbates bullshit and inference-time chain-of-thought (CoT) prompting notably amplify specific bullshit forms, particularly empty rhetoric and paltering. We also observe prevalent machine bullshit in political contexts, with weasel words as the dominant strategy. Our findings highlight systematic challenges in AI alignment and provide new insights toward more truthful LLM behavior.
Citations
Cited by
Discussions
- “While previous work has explored LLM hallucination and sycophancy, we propose machine bullshit as an overarching conceptual framework that can allow researchers to characterize the broader phenomenon [bsky, 36 points, 0 comments]
- You might enjoy this recent scientific paper, then. It characterises the kinds of sycophantic bullshit behaviour found in different Gen AI models - that is, bullshit by Harry Frankfurt’s definition. I [bsky, 4 points, 1 comments]
- Machine Bullshit: Characterizing the Emergent Disregard for Truth in LLMs [hn, 4 points, 1 comments]
- There is literally an academic research paper that technically defines LLM's as "bullshit" machines. Bullshit being an academic term distinct from lies or truth arxiv.org/abs/2507.07484 [bsky, 2 points, 0 comments]
- I love this title of an actual Princeton peer-reviewed study. A good read, following a frustrating two weeks at work where OpenAI models incorrectly performed 57% of the calculations I gave them. arxi [bsky, 1 points, 1 comments]
- arxiv.org/pdf/2507.07484 LLM Bullshit Index from Princeton and Berkeley, includes the proposition of a complementary taxonomy analysing four qualitative forms of bullshit: Empty rhetoric, paltering, w [bsky, 1 points, 0 comments]
- @librarypunk.bsky.social & @libraryfutures.bsky.social, another AI term just dropped - arxiv.org/abs/2507.07484 [bsky, 1 points, 0 comments]
- lots of good work on this so far. Here's an example: arxiv.org/pdf/2507.07484 [bsky, 1 points, 2 comments]
- Machine Bullshit: Characterizing the Emergent Disregard for Truth in LLMs [hn, 1 points, 0 comments]
- useful? Liang, Kaiqu, Haimin Hu, Xuandong Zhao, Dawn Song, Thomas L. Griffiths, and Jaime Fernández Fisac. ‘Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models’. [bsky, 0 points, 0 comments]
- scientific proof: ai is bullshit arxiv.org/abs/2507.07484 [bsky, 0 points, 1 comments]
- The Borrowed Mind by John Nosta I came across this paper "Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models" reading the a pre-publication copy of a book being [bsky, 0 points, 1 comments]
- Another analysis of #ai #bullshit https://arxiv.org/abs/2507.07484v1 [bsky, 0 points, 0 comments]
- Important research from Princeton and Berkeley: "Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models" arxiv.org/abs/2507.07484 [bsky, 0 points, 0 comments]
- While previous work has explored large language model (LLM) hallucination and sycophancy, we propose machine bullshit as an overarching conceptual framework that can allow researchers to characterize [bsky, 0 points, 1 comments]
Related