Why only Micro-F1? Class Weighting of Measures for Relation Classification

2022/05/19 by David Harbecke, Harbecke, David, Yuxuan Chen +5 · 1 citation
Computer Science · #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG) #Machine Learning and Data Classification #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2205.09460

NLP Power! The First Workshop on Efficient Benchmarking in NLP (ACL 2022)

arxiv created 2022/05/19 · openalex publication_date 2022/05/19 · arxiv updated 2022/05/20 · openalex created_date 2022/05/23 · openalex updated_date 2026/07/28

Abstract

Relation classification models are conventionally evaluated using only a single measure, e.g., micro-F1, macro-F1 or AUC. In this work, we analyze weighting schemes, such as micro and macro, for imbalanced datasets. We introduce a framework for weighting schemes, where existing schemes are extremes, and two new intermediate schemes. We show that reporting results of different weighting schemes better highlights strengths and weaknesses of a model.

Cited by

• Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework

Related

• F_β-plot -- a visual tool for evaluating imbalanced data classifiers
• Macro F1 and Macro F1
• A Note on Comparison of F-measures
• On Model Evaluation under Non-constant Class Imbalance
• Metric Learning from Imbalanced Data