2026/03/02 by Stephanie Chen, Oleg Urminsky, Grace Zhang +5 · 1 voice
Computer Science · Medicine · Social Sciences · #AI in Service Interactions #Artificial Intelligence in Healthcare and Education #Ethics and Social Impacts of AI
paper · doi:10.31234/osf.io/xcg26_v1
openalex publication_date 2026/03/02 · openalex created_date 2026/03/03 · openalex updated_date 2026/07/14
Recent research and advances in LLMs have led to widespread concern that AI agents could pose as human online survey-takers. However, it remains unclear how prevalent AI agents are on these platforms and how to effectively detect AI agents. We validated a series of AI detection tests that effectively separated verified-human participants from three AI agents (designed using various prompts). Using these tests, we collected surveys on seven online platforms and find high variance in rates of participants failing AI tests, ranging from 6% to 41% across platforms (compared to a 2.4% in-person human false-positive rate). We demonstrate that undetected AI agents can impact the results of online surveys. Our findings suggest that while there is an urgent need for AI detection tests and consistent, systematic monitoring of data quality on online platforms, currently some platforms seem to provide data with a low rate of AI agents.