Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network

2026/01/01 by Pilar Sánchez-Gijón, Susana Valdez, Sofía Calvo Del Barrio +4
Arts and Humanities · Computer Science · #Translation Studies and Practices #Second Language Learning and Teaching #Text Readability and Simplification

paper · pdf · doi:10.26615/issn.2815-4711.2026_007

Abstract

This paper reports on a collaboration between the Directorate-General for Translation (DGT) and the European Master's in Translation (EMT) to localise the MMLU dataset into 11 European languages.Beyond creating a more inclusive benchmark for LLM evaluation, the project offers master's students authentic, project-based professional training in translation, revision, project management, and multilingual coordination, while highlighting key methodological, administrative, and workflow challenges.

Citations

• AI models collapse when trained on recursively generated data

Related

• Towards Multilingual LLM Evaluation for European Languages
• EuroLLM: Multilingual Language Models for Europe
• Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
• Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
• Findings of the Covid-19 MLIA Machine Translation Task