vix.ing · top · new · best · stats · spec

Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA

2023/03/14 by Benjamin D Douglas, Patrick J. Ewell, Markus Bräuer · 1 voice · 70 citations
Computer Science · Social Sciences · #Privacy-Preserving Technologies in Data #Privacy, Security, and Data Protection #Mobile Crowdsensing and Crowdsourcing

paper · pdf · doi:10.1371/journal.pone.0279720

Abstract

With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher's question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific (1.90) and CloudResearch (2.00) were cheaper than MTurk (4.36) and Qualtrics (8.17). SONA cost 0.00, yet took the longest to collect the data.

Citations

Cited by

Discussions

Related