The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results

2021/10/08 by Marina Fomicheva, Fomicheva, Marina, Piyawat Lertvittayakumjorn +7 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2110.04392

openalex publication_date 2021/10/08 · openalex created_date 2021/10/25 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce the Eval4NLP-2021shared task on explainable quality estimation. Given a source-translation pair, this shared task requires not only to provide a sentence-level score indicating the overall quality of the translation, but also to explain this score by identifying the words that negatively impact translation quality. We present the data, annotation guidelines and evaluation setup of the shared task, describe the six participating systems, and analyze the results. To the best of our knowledge, this is the first shared task on explainable NLP evaluation metrics. Datasets and results are available at https://github.com/eval4nlp/SharedTask2021.

Citations

• Intelligent Agent Transparency in Human–Agent Teaming for Multi-UxV Management
• Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Rethinking Cooperative Rationalization: Introspective Extraction and Complement Control
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Cited by

• Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations

Related

• Towards Explainable Evaluation Metrics for Natural Language Generation
• Revisiting Round-Trip Translation for Quality Estimation
• Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessment
• Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods
• The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics