GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models

2024/06/20 by Tao Zhang, Ziqian Zeng, Zhang, Tao +11 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2406.13925

openalex publication_date 2024/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) are prone to generating content that exhibits gender biases, raising significant ethical concerns. Alignment, the process of fine-tuning LLMs to better align with desired behaviors, is recognized as an effective approach to mitigate gender biases. Although proprietary LLMs have made significant strides in mitigating gender bias, their alignment datasets are not publicly available. The commonly used and publicly available alignment dataset, HH-RLHF, still exhibits gender bias to some extent. There is a lack of publicly available alignment datasets specifically designed to address gender bias. Hence, we developed a new dataset named GenderAlign, aiming at mitigating a comprehensive set of gender biases in LLMs. This dataset comprises 8k single-turn dialogues, each paired with a "chosen" and a "rejected" response. Compared to the "rejected" responses, the "chosen" responses demonstrate lower levels of gender bias and higher quality. Furthermore, we categorized the gender biases in the "rejected" responses of GenderAlign into 4 principal categories. The experimental results show the effectiveness of GenderAlign in reducing gender bias in LLMs.

Cited by

• Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation
• Large Language Models Develop Novel Social Biases Through Adaptive Exploration
• BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responses
• The Fair Game: Auditing & Debiasing AI Algorithms Over Time
• Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

Related

• GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models
• Locating and Mitigating Gender Bias in Large Language Models
• LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
• A database to support the evaluation of gender biases in GPT-4o output