2018/01/02 by Reagan Mozer, Mozer, Reagan, Luke Miratrix +5 · 4 citations
Computer Science · Economics, Econometrics and Finance · #Bayesian Modeling and Causal Inference #Computation and Language (cs.CL) #FOS: Computer and information sciences #Game Theory and Voting Systems #Methodology (stat.ME) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1801.00644
openalex publication_date 2018/01/02 · openalex created_date 2022/10/04 · openalex updated_date 2026/07/28
Matching for causal inference is a well-studied problem, but standard methods\nfail when the units to match are text documents: the high-dimensional and rich\nnature of the data renders exact matching infeasible, causes propensity scores\nto produce incomparable matches, and makes assessing match quality difficult.\nIn this paper, we characterize a framework for matching text documents that\ndecomposes existing methods into: (1) the choice of text representation, and\n(2) the choice of distance metric. We investigate how different choices within\nthis framework affect both the quantity and quality of matches identified\nthrough a systematic multifactor evaluation experiment using human subjects.\nAltogether we evaluate over 100 unique text matching methods along with 5\ncomparison methods taken from the literature. Our experimental results identify\nmethods that generate matches with higher subjective match quality than current\nstate-of-the-art techniques. We enhance the precision of these results by\ndeveloping a predictive model to estimate the match quality of pairs of text\ndocuments as a function of our various distance scores. This model, which we\nfind successfully mimics human judgment, also allows for approximate and\nunsupervised evaluation of new procedures. We then employ the identified best\nmethod to illustrate the utility of text matching in two applications. First,\nwe engage with a substantive debate in the study of media bias by using text\nmatching to control for topic selection when comparing news articles from\nthirteen news sources. We then show how conditioning on text data leads to more\nprecise causal inferences in an observational study examining the effects of a\nmedical intervention.\n