Improving ML Training Data with Gold-Standard Quality Metrics

2025/12/23 by Leslie Barrett, Barrett, Leslie, Michael W. Sherman +1
Decision Sciences · Computer Science · #Reliability and Agreement in Measurement #Imbalanced Data Classification Techniques #Data Mining Algorithms and Applications

paper · pdf · doi:10.48550/arxiv.2512.20577

Abstract

Hand-tagged training data is essential to many machine learning tasks. However, training data quality control has received little attention in the literature, despite data quality varying considerably with the tagging exercise. We propose methods to evaluate and enhance the quality of hand-tagged training data using statistical approaches to measure tagging consistency and agreement. We show that agreement metrics give more reliable results if recorded over multiple iterations of tagging, where declining variance in such recordings is an indicator of increasing data quality. We also show one way a tagging project can collect high-quality training data without requiring multiple tags for every work item, and that a tagger burn-in period may not be sufficient for minimizing tagger errors.

Citations

• CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement
• Inter-Coder Agreement for Computational Linguistics
• The Measurement of Observer Agreement for Categorical Data
• A Coefficient of Agreement for Nominal Scales

Related

• Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data
• Data Quality Toolkit: Automatic assessment of data quality and remediation for machine learning datasets
• Learning to Rank from Samples of Variable Quality
• Quantity vs Quality: Investigating the Trade-Off between Sample Size and Label Reliability
• Quality of Data in Machine Learning