Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias

2022/08/31 by Wood, Nick, Sumit Sidana, Sidana, Sumit
Computer Science · Decision Sciences · Economics, Econometrics and Finance · #Auction Theory and Applications #D.2.4 #FOS: Computer and information sciences #Game Theory and Voting Systems #H.3.3 #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Recommender Systems and Techniques

paper · pdf · doi:10.48550/arxiv.2208.14980

openalex publication_date 2022/08/31 · openalex created_date 2023/02/12 · openalex updated_date 2026/07/28

Abstract

In this work, we present a novel way of computing IPS using a position-bias model for deterministic logging policies. This technique significantly widens the policies on which OPE can be used. We validate this technique using two different experiments on industry-scale data. The OPE results are clearly strongly correlated with the online results, with some constant bias. The estimator requires the examination model to be a reasonably accurate approximation of real user behaviour.

Related

• LLMs for estimating positional bias in logged interaction data
• Offline Evaluation of Ranked Lists using Parametric Estimation of Propensities
• Offline Comparison of Ranking Functions using Randomized Data
• Unconfounded Propensity Estimation for Unbiased Ranking
• Consistent Position Bias Estimation without Online Interventions for Learning-to-Rank