Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

2023/06/13 by Veniamin Veselovsky, Manoel Horta Ribeiro, Veselovsky, Veniamin +3 · 12 voices · 11 citations
Computer Science · #Mobile Crowdsensing and Crowdsourcing #AI in Service Interactions #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2306.07899

Abstract

Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: https://github.com/epfl-dlab/GPTurk

Cited by

• Posts of Peril: Detecting Information About Hazards in Text
• LLM Social Simulations Are a Promising Research Method
• Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact
• A Customer Journey in the Land of Oz: Leveraging the Wizard of Oz Technique to Model Emotions in Customer Service Interactions
• The differential effects of metaphor on comprehensibility and comprehension of environmental concepts
• Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writers
• Learning in an Echo Chamber: Online Learning with Replay Adversary
• The Ouroboros effect and heterodox domains
• Lessons from complex systems science for AI governance
• Synthesizing Attitudes, Predicting Actions (SAPA): Behavioral Theory-Guided LLMs for Ridesourcing Mode Choice Modeling
• Network formation and dynamics among multi-LLMs

Discussions

• 33-46% of workers on MTurk used LLMs in a text production task [hn, 189 points, 133 comments]
• «We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crow [bsky, 10 points, 0 comments]
• In fact, they are doing exactly that: arxiv.org/abs/2306.07899 [bsky, 6 points, 1 comments]
• From a preprint in June last year “through a combination of keystroke detection and synthetic text classification, [we] estimate that 33-46% of crowd workers used LLMs when completing the task” arxiv [bsky, 4 points, 0 comments]
• Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks [lemmy, 2 points, 0 comments]
• “We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crow [bsky, 1 points, 0 comments]
• Crowd-source workers using LLMs 30-40% of the time arxiv.org/abs/2306.07899 I'm dying [bsky, 1 points, 0 comments]
• arxiv.org/abs/2306.07899 [bsky, 1 points, 0 comments]
• If I follow correctly, this paper suggests that Amazon Mechanical Turk workers are (reasonably) relying increasingly on LLMs to generate input that researchers might assume is actually human-generated [bsky, 1 points, 1 comments]
• and oh no: https://arxiv.org/abs/2306.07899 [bsky, 1 points, 0 comments]
• A lot of #LLM training projects are using mechanical turks... And those people are using #ChatGPT to generate the answers :D Ouroboros https://arxiv.org/abs/2306.07899v1?utm_medium=social&utm_source=t [bsky, 0 points, 0 comments]
• Some recent papers I’m proud of include quasi-experiments on the effects of deplatforming Parler (PNASNexus) and on removing comments on Facebook (WWW). In a more recent project, I’m examining the imp [bsky, 0 points, 1 comments]

Related

• Prevalence and prevention of large language model use in crowd work
• Power-up! What Can Generative Models Do for Human Computation Workflows?
• Are Large Language Models the future crowd workers of Linguistics?
• Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth
• Revisiting Prompt Engineering via Declarative Crowdsourcing