2021/02/16 by Robert McCluskey, Amir Enshaei, McCluskey, Robert +4 · 3 citations
Computer Science · Engineering · Mathematics · #Artificial Intelligence (cs.AI) #Artificial intelligence #Binomial distribution #Computer science #Data Stream Mining Techniques #Data mining #Data science #Econometrics #Engineering #FOS: Computer and information sciences #Ground truth #Inference #Machine Learning and Data Classification #Machine learning #Mathematics #Mobile Crowdsensing and Crowdsourcing #Operations research #Sample (material) #Scale (ratio) #Statistics #Task (project management) #cs.AI
paper · pdf · doi:10.48550/arxiv.2102.08482
published in arXiv (Cornell University) (Cornell University) · 16 pages, 5 figures
arxiv created 2021/02/16 · openalex publication_date 2021/02/16 · arxiv updated 2021/02/18 · openalex created_date 2022/07/25 · openalex updated_date 2026/08/06
Employing multiple workers to label data for machine learning models has\nbecome increasingly important in recent years with greater demand to collect\nhuge volumes of labelled data to train complex models while mitigating the risk\nof incorrect and noisy labelling. Whether it is large scale data gathering on\npopular crowd-sourcing platforms or smaller sets of workers in high-expertise\nlabelling exercises, there are various methods recommended to gather a\nconsensus from employed workers and establish ground-truth labels. However,\nthere is very little research on how the various parameters of a labelling task\ncan impact said methods. These parameters include the number of workers, worker\nexpertise, number of labels in a taxonomy and sample size. In this paper,\nMajority Vote, CrowdTruth and Binomial Expectation Maximisation are\ninvestigated against the permutations of these parameters in order to provide\nbetter understanding of the parameter settings to give an advantage in\nground-truth inference. Findings show that both Expectation Maximisation and\nCrowdTruth are only likely to give an advantage over majority vote under\ncertain parameter conditions, while there are many cases where the methods can\nbe shown to have no major impact. Guidance is given as to what parameters\nmethods work best under, while the experimental framework provides a way of\ntesting other established methods and also testing new methods that can attempt\nto provide advantageous performance where the methods in this paper did not. A\ngreater level of understanding regarding optimal crowd-sourcing parameters is\nalso achieved.\n