2013/09/30 by Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney · 2 citations
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Coding (social sciences) #Computer science #Computer security #Crowds #Crowdsourcing #Data Stream Mining Techniques #Data mining #Data science #Decoding methods #Machine learning #Majority rule #Mathematics #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data #Voting #World Wide Web #cs.IT #cs.SI #math.IT
paper · pdf · doi:10.1109/jstsp.2014.2316116
20 pages, 11 figures, under revision, IEEE Journal of Selected Topics in Signal Processing
arxiv created 2014/01/22 · openalex publication_date 2014/04/08 · arxiv updated 2015/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Crowdsourcing systems often have crowd workers that perform unreliable work on the task they are assigned. In this paper, we propose the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification despite unreliable crowd workers. Coding theory based techniques also allow us to pose easy-to-answer binary questions to the crowd workers. We consider three different crowdsourcing models: systems with independent crowd workers, systems with peer-dependent reward schemes, and systems where workers have common sources of information. For each of these models, we analyze classification performance with the proposed coding-based scheme. We develop an ordering principle for the quality of crowds and describe how system performance changes with the quality of the crowd. We also show that pairing among workers and diversification of the questions help in improving system performance. We demonstrate the effectiveness of the proposed coding-based scheme using both simulated data and real datasets from Amazon Mechanical Turk, a crowdsourcing microtask platform. Results suggest that use of good codes may improve the performance of the crowdsourcing task over typical majority-voting approaches.