2024/12/04 by Kyle L. Marquardt, Daniel Pemstein, Constanza Sanhueza Petrarca +5 · 1 voice · 3 citations
Decision Sciences · Social Sciences · #Artificial intelligence #Auction Theory and Applications #Bounding overwatch #Coding (social sciences) #Computer science #Computer security #Context (archaeology) #Crowds #Data science #Media Influence and Politics #Misinformation and Its Impacts #Scope (computer science) #Sociology #Theridiidae #Typology
paper · doi:10.1177/01925121241293459
published in International Political Science Review 46(4), 622-638 (SAGE Publishing)
openalex publication_date 2024/12/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
Political scientists increasingly use crowdworkers to produce data, predominantly in the context of coding researcher-curated text or to retrieve simple data from the internet. In this article, we provide a theoretical and empirical basis for understanding when crowdworkers can provide data of sufficient quality to substitute for other types of coders. First, we introduce a typology of data-producing actors – experts, trained coders and crowds – and hypothesize factors that affect the substitutability of crowdworkers. We then examine how crowdworkers perform across coding tasks that vary along multiple dimensions of difficulty: information verifiability, availability and complexity. The results provide scope conditions bounding the substitutability of crowdworkers in political science applications. Although crowds can substitute for trained coders in the context of relatively simple information retrieval tasks, there is little evidence that crowdworkers can substitute for experts, whose tasks require both information retrieval and data synthesis.