Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks
2023/06/13 by Veniamin Veselovsky, Manoel Horta Ribeiro, Veselovsky, Veniamin +3 · 12 voices · 73 citations
Computer Science · Engineering · #AI in Service Interactions #Artificial intelligence #Automatic summarization #Code (set theory) #Computer science #Computer security #Crowdsourcing #Data science #Engineering #Human intelligence #Keystroke logging #Machine learning #Mobile Crowdsensing and Crowdsourcing #Natural language processing #Task (project management) #Text Readability and Simplification #World Wide Web
paper · pdf · doi:10.48550/arxiv.2306.07899
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/06/13 · openalex created_date 2023/06/15 · openalex updated_date 2026/07/28
Abstract
Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: https://github.com/epfl-dlab/GPTurk
Cited by
Discussions
- 33-46% of workers on MTurk used LLMs in a text production task [hn, 189 points, 133 comments]
- «We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crow [bsky, 10 points, 0 comments]
- In fact, they are doing exactly that: arxiv.org/abs/2306.07899 [bsky, 6 points, 1 comments]
- From a preprint in June last year “through a combination of keystroke detection and synthetic text classification, [we] estimate that 33-46% of crowd workers used LLMs when completing the task” arxiv [bsky, 4 points, 0 comments]
- Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks [lemmy, 2 points, 0 comments]
- “We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crow [bsky, 1 points, 0 comments]
- Crowd-source workers using LLMs 30-40% of the time arxiv.org/abs/2306.07899 I'm dying [bsky, 1 points, 0 comments]
- arxiv.org/abs/2306.07899 [bsky, 1 points, 0 comments]
- If I follow correctly, this paper suggests that Amazon Mechanical Turk workers are (reasonably) relying increasingly on LLMs to generate input that researchers might assume is actually human-generated [bsky, 1 points, 1 comments]
- and oh no: https://arxiv.org/abs/2306.07899 [bsky, 1 points, 0 comments]
- A lot of #LLM training projects are using mechanical turks... And those people are using #ChatGPT to generate the answers :D Ouroboros https://arxiv.org/abs/2306.07899v1?utm_medium=social&utm_source=t [bsky, 0 points, 0 comments]
- Some recent papers I’m proud of include quasi-experiments on the effects of deplatforming Parler (PNASNexus) and on removing comments on Facebook (WWW). In a more recent project, I’m examining the imp [bsky, 0 points, 1 comments]
Related