2018/02/01 by Mucahid Kutlu, Mücahid Kutlu, Kutlu, Mucahid +4
Computer Science · Decision Sciences · Mathematics · #Artificial intelligence #Computer science #Correlation #Data Quality and Management #Data mining #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Information Retrieval and Search Behavior #Information retrieval #Mathematics #Statistics #Web Data Mining and Analysis #cs.IR
paper · pdf · doi:10.48550/arxiv.1802.00323
arxiv created 2018/02/01 · openalex publication_date 2018/02/01 · arxiv updated 2018/02/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Because researchers typically do not have the time or space to present more than a few evaluation metrics in any published study, it can be difficult to assess relative effectiveness of prior methods for unreported metrics when baselining a new method or conducting a systematic meta-review. While sharing of study data would help alleviate this, recent attempts to encourage consistent sharing have been largely unsuccessful. Instead, we propose to enable relative comparisons with prior work across arbitrary metrics by predicting unreported metrics given one or more reported metrics. In addition, we further investigate prediction of high-cost evaluation measures using low-cost measures as a potential strategy for reducing evaluation cost. We begin by assessing the correlation between 23 IR metrics using 8 TREC test collections. Measuring prediction error wrt. R-square and Kendall's tau, we show that accurate prediction of MAP, P@10, and RBP can be achieved using only 2-3 other metrics. With regard to lowering evaluation cost, we show that RBP(p=0.95) can be predicted with high accuracy using measures with only evaluation depth of 30. Taken together, our findings provide a valuable proof-of-concept which we expect to spur follow-on work by others in proposing more sophisticated models for metric prediction.