2022/09/17 by Sanne Vrijenhoek, G. F. Benedict, Vrijenhoek, Sanne +7 · 6 citations
Physics and Astronomy · Social Sciences · #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Media Influence and Politics #Misinformation and Its Impacts #Opinion Dynamics and Social Influence
paper · pdf · doi:10.48550/arxiv.2209.13520
openalex publication_date 2022/09/17 · openalex created_date 2022/10/01 · openalex updated_date 2026/07/28
In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models. However, this is not expressive of the social science's interpretation of diversity, which accounts for a news organization's norms and values and which we here refer to as normative diversity. We introduce RADio, a versatile metrics framework to evaluate recommendations according to these normative goals. RADio introduces a rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a user's decreasing propensity to observe items further down a list and (ii) full distributional shifts as opposed to point estimates. We evaluate RADio's ability to reflect five normative concepts in news recommendations on the Microsoft News Dataset and six (neural) recommendation algorithms, with the help of our metadata enrichment pipeline. We find that RADio provides insightful estimates that can potentially be used to inform news recommender system design.