2012/04/16 by Nicolás Della Penna, Della Penna, Nicolás, Mark D. Reid +1
Computer Science · #Computer Science and Game Theory (cs.GT) #FOS: Computer and information sciences #Social and Information Networks (cs.SI) #cs.GT #cs.SI
paper · pdf · doi:10.48550/arxiv.1204.3511
Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991)
arxiv created 2012/04/16 · arxiv updated 2012/04/17
A common use of crowd sourcing is to obtain labels for a dataset. Several algorithms have been proposed to identify uninformative members of the crowd so that their labels can be disregarded and the cost of paying them avoided. One common motivation of these algorithms is to try and do without any initial set of trusted labeled data. We analyse this class of algorithms as mechanisms in a game-theoretic setting to understand the incentives they create for workers. We find an impossibility result that without any ground truth, and when workers have access to commonly shared 'prejudices' upon which they agree but are not informative of true labels, there is always equilibria where all agents report the prejudice. A small amount amount of gold standard data is found to be sufficient to rule out these equilibria.