2013/04/30 by Balaji Lakshminarayanan, Yee Whye Teh, Lakshminarayanan, Balaji +1 · 1 citation
Computer Science · Mathematics · #Data Stream Mining Techniques #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mobile Crowdsensing and Crowdsourcing #Privacy-Preserving Technologies in Data #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1305.0015
arxiv created 2013/04/30 · openalex publication_date 2013/04/30 · arxiv updated 2013/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal data have been proposed mostly as extensions of their binary/categorical counterparts and have received little attention in the crowdsourcing literature. We propose a new model for crowdsourced ordinal data that accounts for instance difficulty as well as annotator expertise, and derive a variational Bayesian inference algorithm for parameter estimation. We analyze the ordinal extensions of several state-of-the-art annotator models for binary/categorical labels and evaluate the performance of all the models on two real world datasets containing ordinal query-URL relevance scores, collected through Amazon's Mechanical Turk. Our results indicate that the proposed model performs better or as well as existing state-of-the-art methods and is more resistant to `spammy' annotators (i.e., annotators who assign labels randomly without actually looking at the instance) than popular baselines such as mean, median, and majority vote which do not account for annotator expertise.