2013/09/12 by Arthur Carvalho, Stanko Dimitrov, Carvalho, Arthur +3
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Digital Libraries (cs.DL) #Expert finding and Q&A systems #FOS: Computer and information sciences #FOS: Mathematics #Mobile Crowdsensing and Crowdsourcing #Multiagent Systems (cs.MA) #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.1309.3197
openalex publication_date 2013/09/12 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28
When eliciting opinions from a group of experts, traditional devices used to\npromote honest reporting assume that there is an observable future outcome. In\npractice, however, this assumption is not always reasonable. In this paper, we\npropose a scoring method built on strictly proper scoring rules to induce\nhonest reporting without assuming observable outcomes. Our method provides\nscores based on pairwise comparisons between the reports made by each pair of\nexperts in the group. For ease of exposition, we introduce our scoring method\nby illustrating its application to the peer-review process. In order to do so,\nwe start by modeling the peer-review process using a Bayesian model where the\nuncertainty regarding the quality of the manuscript is taken into account.\nThereafter, we introduce our scoring method to evaluate the reported reviews.\nUnder the assumptions that reviewers are Bayesian decision-makers and that they\ncannot influence the reviews of other reviewers, we show that risk-neutral\nreviewers strictly maximize their expected scores by honestly disclosing their\nreviews. We also show how the group's scores can be used to find a consensual\nreview. Experimental results show that encouraging honest reporting through the\nproposed scoring method creates more accurate reviews than the traditional\npeer-review process.\n