Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP

2023/02/11 by Han, Xudong, Baldwin, Timothy, Cohn, Trevor · 1 citation
#Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2302.05711

Abstract

Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct. However current progress is hampered by a plurality of definitions of bias, means of quantification, and oftentimes vague relation between debiasing algorithms and theoretical measures of bias. This paper seeks to clarify the current situation and plot a course for meaningful progress in fair learning, with two key contributions: (1) making clear inter-relations among the current gamut of methods, and their relation to fairness theory; and (2) addressing the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research. Putting them together, we make several recommendations to help shape future work.

Cited by

• On the Interplay between Human Label Variation and Model Fairness

Related

• Fairness Certification for Natural Language Processing and Large Language Models
• Advancing Fairness in Natural Language Processing: From Traditional Methods to Explainability
• Quantifying Social Biases in NLP: A Generalization and Empirical\n Comparison of Extrinsic Fairness Metrics
• A Survey on Bias and Fairness in Natural Language Processing
• Fairness Definitions in Language Models Explained