2016/05/19 by Qianqian Xu, Xu, Qianqian, Jiechao Xiong +5 · 1 citation
Computer Science · Decision Sciences · #Anomaly Detection Techniques and Applications #Auction Theory and Applications #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (stat.ML) #Mobile Crowdsensing and Crowdsourcing
paper · pdf · doi:10.48550/arxiv.1605.05860
openalex publication_date 2016/05/19 · openalex created_date 2019/07/30 · openalex updated_date 2026/07/28
With the rapid growth of crowdsourcing platforms it has become easy and\nrelatively inexpensive to collect a dataset labeled by multiple annotators in a\nshort time. However due to the lack of control over the quality of the\nannotators, some abnormal annotators may be affected by position bias which can\npotentially degrade the quality of the final consensus labels. In this paper we\nintroduce a statistical framework to model and detect annotator's position bias\nin order to control the false discovery rate (FDR) without a prior knowledge on\nthe amount of biased annotators - the expected fraction of false discoveries\namong all discoveries being not too high, in order to assure that most of the\ndiscoveries are indeed true and replicable. The key technical development\nrelies on some new knockoff filters adapted to our problem and new algorithms\nbased on the Inverse Scale Space dynamics whose discretization is potentially\nsuitable for large scale crowdsourcing data analysis. Our studies are supported\nby experiments with both simulated examples and real-world data. The proposed\nframework provides us a useful tool for quantitatively studying annotator's\nabnormal behavior in crowdsourcing data arising from machine learning,\nsociology, computer vision, multimedia, etc.\n