2018/12/12 by Aman Agarwal, Ivan Zaitsev, Xuanhui Wang +3 · 3 citations
Computer Science · #Counterfactual thinking #Estimator #Expert finding and Q&A systems #Information Retrieval and Search Behavior #Machine Learning and Algorithms #Position (finance) #Presentation (obstetrics) #Propensity score matching #Range (aeronautics) #Ranking (information retrieval) #Relevance (law) #cs.IR
paper · pdf · doi:10.1145/3289600.3291017
arxiv created 2018/12/12 · arxiv updated 2018/12/14 · openalex created_date 2018/12/22 · openalex publication_date 2019/01/30 · openalex updated_date 2026/08/06
Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal. While it was recently shown how counterfactual learning-to-rank (LTR) approaches \citeJoachims/etal/17a can provably overcome presentation bias when observation propensities are known, it remains to show how to effectively estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model. Second, we propose a new extremum estimator that makes effective use of this data. In an empirical evaluation, we find that the new estimator provides superior propensity estimates in two real-world systems -- Arxiv Full-text Search and Google Drive Search. Beyond these two points, we find that the method is robust to a wide range of settings in simulation studies.