2024/09/05 by Olivia Sturman, Aparna Joshi, Sturman, Olivia +7 · 1 citation
Business, Management and Accounting · Computer Science · #Artificial Intelligence (cs.AI) #Blood donation and transfusion practices #Computation and Language (cs.CL) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection #Machine Learning (cs.LG) #Spam and Phishing Detection
paper · pdf · doi:10.48550/arxiv.2409.13705
openalex publication_date 2024/09/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Increasing use of large language models (LLMs) demand performant guardrails to ensure the safety of inputs and outputs of LLMs. When these safeguards are trained on imbalanced data, they can learn the societal biases. We present a light-weight, post-processing method for mitigating counterfactual fairness in closed-source text safety classifiers. Our approach involves building an ensemble that not only outperforms the input classifiers and policy-aligns them, but also acts as a debiasing regularizer. We introduce two threshold-agnostic metrics to assess the counterfactual fairness of a model, and demonstrate how combining these metrics with Fair Data Reweighting (FDW) helps mitigate biases. We create an expanded Open AI dataset, and a new templated LLM-generated dataset based on user-prompts, both of which are counterfactually balanced across identity groups and cover four key areas of safety; we will work towards publicly releasing these datasets. Our results show that our approach improves counterfactual fairness with minimal impact on model performance.