2022/12/19 by Chengwen Wang, Wang, Chengwen, Qingxiu Dong +7 · 1 citation
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Machine Learning in Healthcare #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2212.09272
arxiv created 2022/12/19 · openalex publication_date 2022/12/19 · arxiv updated 2022/12/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Datasets serve as crucial training resources and model performance trackers. However, existing datasets have exposed a plethora of problems, inducing biased models and unreliable evaluation results. In this paper, we propose a model-agnostic dataset evaluation framework for automatic dataset quality evaluation. We seek the statistical properties of the datasets and address three fundamental dimensions: reliability, difficulty, and validity, following a classical testing theory. Taking the Named Entity Recognition (NER) datasets as a case study, we introduce 9 statistical metrics for a statistical dataset evaluation framework. Experimental results and human evaluation validate that our evaluation framework effectively assesses various aspects of the dataset quality. Furthermore, we study how the dataset scores on our statistical metrics affect the model performance, and appeal for dataset quality evaluation or targeted dataset improvement before training or testing models.