Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval

2023/11/03 by Yang, Jinrui, Baldwin, Timothy, Cohn, Trevor · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2311.01870

Abstract

We present Multi-EuP, a new multilingual benchmark dataset, comprising 22K multi-lingual documents collected from the European Parliament, spanning 24 languages. This dataset is designed to investigate fairness in a multilingual information retrieval (IR) context to analyze both language and demographic bias in a ranking context. It boasts an authentic multilingual corpus, featuring topics translated into all 24 languages, as well as cross-lingual relevance judgments. Furthermore, it offers rich demographic information associated with its documents, facilitating the study of demographic bias. We report the effectiveness of Multi-EuP for benchmarking both monolingual and multilingual IR. We also conduct a preliminary experiment on language bias caused by the choice of tokenization strategy.

Cited by

• Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset
• Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods
• Benchmarking Gender and Political Bias in Large Language Models

Related

• Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset
• Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods
• Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness
• Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection
• NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval