Human Uncertainty and Ranking Error -- The Secret of Successful\n Evaluation in Predictive Data Mining

2017/08/17 by Kevin Jasberg, Jasberg, Kevin, Sergej Sizov +1
Computer Science · #Bayesian Modeling and Causal Inference #Data Management and Algorithms

paper · pdf · doi:10.48550/arxiv.1708.05688

Abstract

One of the most crucial issues in data mining is to model human behaviour in\norder to provide personalisation, adaptation and recommendation. This usually\ninvolves implicit or explicit knowledge, either by observing user interactions,\nor by asking users directly. But these sources of information are always\nsubject to the volatility of human decisions, making utilised data uncertain to\na particular extent. In this contribution, we elaborate on the impact of this\nhuman uncertainty when it comes to comparative assessments of different data\nmining approaches. In particular, we reveal two problems: (1) biasing effects\non various metrics of model-based prediction and (2) the propagation of\nuncertainty and its thus induced error probabilities for algorithm rankings.\nFor this purpose, we introduce a probabilistic view and prove the existence of\nthose problems mathematically, as well as provide possible solution strategies.\nWe exemplify our theory mainly in the context of recommender systems along with\nthe metric RMSE as a prominent example of precision quality measures.\n

Related

• Assessment of Prediction Techniques: The Impact of Human Uncertainty
• Probabilistic Perspectives on Collecting Human Uncertainty in Predictive Data Mining
• Re-Evaluating the Netflix Prize - Human Uncertainty and its Impact on\n Reliability
• Are Recommenders Self-Aware? Label-Free Recommendation Performance Estimation via Model Uncertainty
• Mitigating Exploitation Bias in Learning to Rank with an Uncertainty-aware Empirical Bayes Approach