Evaluating Large Language Models through Gender and Racial Stereotypes

2023/11/24 by Malik, Ananya
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2311.14788

Abstract

Language Models have ushered a new age of AI gaining traction within the NLP community as well as amongst the general population. AI's ability to make predictions, generations and its applications in sensitive decision-making scenarios, makes it even more important to study these models for possible biases that may exist and that can be exaggerated. We conduct a quality comparative study and establish a framework to evaluate language models under the premise of two kinds of biases: gender and race, in a professional setting. We find out that while gender bias has reduced immensely in newer models, as compared to older ones, racial bias still exists.

Related

• Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications
• Addressing Stereotypes in Large Language Models: A Critical Examination and Mitigation
• Gender bias and stereotypes in Large Language Models
• LLM Bias Evaluation: Gender, Racial, and Age Disparities in Occupational and Crime Scenarios
• More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models