2018/06/13 by Jingyan Wang, Nihar B. Shah, Wang, Jingyan +1 · 6 citations
Business, Management and Accounting · Computer Science · Decision Sciences · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Forecasting Techniques and Applications #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Organizational Management and Leadership #Stock Market Forecasting Methods #cs.AI #cs.IT #cs.LG #math.IT #stat.ML
paper · pdf · doi:10.48550/arxiv.1806.05085
openalex publication_date 2018/06/13 · arxiv created 2018/09/13 · arxiv updated 2018/09/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood. In the absence of simplifying assumptions on the miscalibration, it is widely believed by the crowdsourcing community that the only useful information in the cardinal scores is the induced ranking. In this paper, inspired by the framework of Stein's shrinkage, empirical Bayes, and the classic two-envelope problem, we contest this widespread belief. Specifically, we consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations which are only required to be consistent with the induced ranking. We design estimators which despite making no assumptions on the miscalibration, strictly and uniformly outperform all possible estimators that rely on only the ranking. Our estimators are flexible in that they can be used as a plug-in for a variety of applications, and we provide a proof-of-concept for A/B testing and ranking. Our results thus provide novel insights in the eternal debate between cardinal and ordinal data.