2022/12/20 by Lining Zhang, Zhang, Lining, Simon Mille +19 · 1 citation
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Mobile Crowdsensing and Crowdsourcing #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2212.10397
openalex publication_date 2022/12/20 · openalex created_date 2023/01/04 · openalex updated_date 2026/08/04
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.