Interpretable Multi-dataset Evaluation for Named Entity Recognition

2020/11/13 by Jinlan Fu, Pengfei Liu, Fu, Jinlan +3
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2011.06854

openalex publication_date 2020/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: https://github.com/neulab/InterpretEval.

Citations

• Relative Power of the Wilcoxon Test, the Friedman Test, and Repeated-Measures ANOVA on Ranks
• RethinkCWS: Is Chinese Word Segmentation a Solved Task?

Related

• What do we Really Know about State of the Art NER?
• Cascaded Models for Better Fine-Grained Named Entity Recognition
• What Matters When Building Universal Multilingual Named Entity Recognition Models?
• Evaluating Named Entity Recognition Using Few-Shot Prompting with Large Language Models
• Is 'Hope' a person or an idea? A pilot benchmark for NER: comparing traditional NLP tools and large language models on ambiguous entities