2022/06/21 by Walid Krichene, Steffen Rendle · 69 citations
Business, Management and Accounting · Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Algorithm #Computation #Computer science #Consumer Market Behavior and Pricing #Data mining #Information retrieval #Machine learning #Mathematics #Metric (unit) #Quality (philosophy) #Recommender Systems and Techniques #Recommender system #Sample (material) #Sample size determination #Sampling (signal processing) #Set (abstract data type) #Statistics
paper · pdf · doi:10.1145/3535335
published in Communications of the ACM 65(7), 75-83 (Association for Computing Machinery)
openalex publication_date 2022/06/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/25
Recommender systems personalize content by recommending items to users. Item recommendation algorithms are evaluated by metrics that compare the positions of truly relevant items among the recommended items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates such sampled metrics in more detail and shows that they are inconsistent with their exact counterpart, in the sense that they do not persist relative statements, for example, recommender A is better than B , not even in expectation. Moreover, the smaller the sample size, the less difference there is between metrics, and for very small sample size, all metrics collapse to the AUC metric. We show that it is possible to improve the quality of the sampled metrics by applying a correction, obtained by minimizing different criteria. We conclude with an empirical evaluation of the naive sampled metrics and their corrected variants. To summarize, our work suggests that sampling should be avoided for metric calculation, however if an experimental study needs to sample, the proposed corrections can improve the quality of the estimate.