AI generates covertly racist decisions about people based on their dialect
2024/08/28 by Valentin Hofmann, Pratyusha Kalluri, Dan Jurafsky +1 · 2 voices · 86 citations
Social Sciences · Computer Science · #Artificial Intelligence in Law #Topic Modeling #Natural Language Processing Techniques
paper · pdf · doi:10.1038/s41586-024-07856-5
Abstract
Abstract Hundreds of millions of people now interact with language models, with uses ranging from help with writing 1,2 to informing hiring decisions 3 . However, these language models are known to perpetuate systematic racial prejudices, making their judgements biased in problematic ways about groups such as African Americans 4–7 . Although previous research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time, particularly in the United States after the civil rights movement 8,9 . It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice, exhibiting raciolinguistic stereotypes about speakers of African American English (AAE) that are more negative than any human stereotypes about African Americans ever experimentally recorded. By contrast, the language models’ overt stereotypes about African Americans are more positive. Dialect prejudice has the potential for harmful consequences: language models are more likely to suggest that speakers of AAE be assigned less-prestigious jobs, be convicted of crimes and be sentenced to death. Finally, we show that current practices of alleviating racial bias in language models, such as human preference alignment, exacerbate the discrepancy between covert and overt stereotypes, by superficially obscuring the racism that language models maintain on a deeper level. Our findings have far-reaching implications for the fair and safe use of language technology.
Citations
Cited by
- Large Language Models Are Biased Because They Are Large Language Models
- Multimodal large language models can make context-sensitive hate speech evaluations aligned with human judgement
- Training large language models on narrow tasks can lead to broad misalignment
- A Framework for Facilitating Young People’s Sociocritical AI Literacies
- Socially prescriptive speech technologies: Linguistic, technical, and ethical issues
- AI as a talent management tool: An organizational justice perspective
- Ethical Considerations in the Deployment of Artificial Intelligence in Surgery
- AI-Powered Browsers Are Broadly Accurate News Summarizers That Reduce Political Bias and Negative Affect
- Language Models Embody and Amplify Human Cognitive Distortions: What Is to Be Done?
- An approach to systemic risks of AI through the lens of emergence, collective action problems, and externalities
- Data and trained models for "Empirical Evidence of Large Language Model's Influence on Human Spoken Communication"
- Algorithmic Monocultures in Hiring
- A conceptual framework for ideology beyond the left and right
- Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation
- The Digital Divide in Generative AI: Evidence from Large Language Model Use in College Admissions Essays
- Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
- A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
- Rejected Dialects: Biases Against African American Language in Reward Models
- Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models
- Biased AI writing assistants shift users’ attitudes on societal issues
- Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
- Responsible Intelligence in Practice: A Fairness Audit of Open Large Language Models for Library Reference Services
- Impacts of Racial Bias in Historical Training Data for News AI
- Linear socio-demographic representations emerge in Large Language Models from indirect cues
- Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context
- Us-vs-Them bias in Large Language Models
- Understanding Down Syndrome Stereotypes in LLM-Based Personas
- Writing in Symbiosis: Mapping Human Creative Agency in the AI Era
- Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses
- Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
- AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys
- Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiency
- Large Language Models Develop Novel Social Biases Through Adaptive Exploration
- Why Public Service AI Governance Frameworks Risk Failing in the Age of General-Purpose AI: Lessons from Policing
- Exploring the Intersection of AI, Language, and Law: A Bibliometric Analysis
- Large language models, social demography, and hegemony: comparing authorship in human and synthetic text
- Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks
- DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English
- End-of-life education: An explorative study using artificial intelligence simulations in undergraduate nursing
- Public Opinion on the Politics of AI Alignment: Cross-National Evidence on Expectations for AI Moderation From Germany and the United States
- Exploring LLMs for Automated Generation and Adaptation of Questionnaires
- Identity-Aware Large Language Models require Cultural Reasoning
- Using natural language processing to analyse text data in behavioural science
- The Social Cost of Intelligence: Emergence, Propagation, and Amplification of Stereotypical Bias in Multi-Agent Systems
- Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions
- Assessing Human Rights Risks in AI: A Framework for Model Evaluation
- Disclosure and Evaluation as Fairness Interventions for General-Purpose AI
- Mitigating Biases in Language Models via Bias Unlearning
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Responsible AI in Marketing: AI Booing and AI Washing Cycle of AI Mistrust
- Linguistic Affordances Framework: A Linguistic-Sociological Approach for the Social Study of Language Technology
- Generative AI and linguistic diversity in academic writing and publishing
- AI systems and the reproduction of (standard) language ideologies in World Englishes
- AI as moral cover: How algorithmic bias exploits psychological mechanisms to perpetuate social inequality
- Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset
- Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases
- Designing Effective AI Explanations for Misinformation Detection: A Comparative Study of Content, Social, and Combined Explanations
- AI reasoning effort predicts human decision time in content moderation
- Ask ChatGPT: Caveats and Mitigations for Individual Users of AI Chatbots
- Bias Association Discovery Framework for Open-Ended LLM Generations
- EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Norms
- Generative language models exhibit social identity biases. [europepmc]
- Modelling the impact of environmental and social determinants on mental health using generative agents. [europepmc]
- Current applications and challenges in large language models for patient care: a systematic review. [europepmc]
- The sociolinguistic foundations of language modeling. [europepmc]
- Gender and racial bias issues in a commercial "tone of voice" analysis system. [europepmc]
- Factors modulating perception and production of speech by AI tools: a test case of Amazon Alexa and Polly. [europepmc]
- Public Perceptions of Judges' Use of AI Tools in Courtroom Decision-Making: An Examination of Legitimacy, Fairness, Trust, and Procedural Justice. [europepmc]
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias. [europepmc]
- Computational challenges arising in algorithmic fairness and health equity with generative AI. [europepmc]
- Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models. [europepmc]
- Examining Chat GPT with nonwords and machine psycholinguistic techniques. [europepmc]
- Embodied artificial intelligence in ophthalmology. [europepmc]
- Biased echoes: Large language models reinforce investment biases and increase portfolio risks of private investors. [europepmc]
- School-Based Online Surveillance of Youth: Systematic Search and Content Analysis of Surveillance Company Websites. [europepmc]
- AI-AI bias: Large language models favor communications generated by large language models. [europepmc]
- Artificial Intelligence in migrant health: a critical perspective on opportunities and risks. [europepmc]
- AI biases as asymmetries: a review to guide practice. [europepmc]
- Training large language models on narrow tasks can lead to broad misalignment. [europepmc]
- Acceptability of Using Large Language Models to Support Smoking Cessation Attempts: a Qualitative Study Among African Americans Who Smoke. [europepmc]
- Evaluation of cross-ethnic emotion recognition capabilities in multimodal large language models using the reading the mind in the eyes test. [europepmc]
- Biased AI writing assistants shift users' attitudes on societal issues. [europepmc]
- Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI. [europepmc]
- Child and adolescent psychiatry: challenges, solutions, opportunities, and future directions. [europepmc]
- Algorithmic bias [wikipedia]
Discussions
- This is the study she’s referencing. Hofmann, V., Kalluri, P.R., Jurafsky, D. et al. (2024) AI generates covertly racist decisions about people based on their dialect. Nature. doi.org/10.1038/s415... [bsky, 31 points, 1 comments]
- Also, the affirmation bot encodes the strongest covert anti-Black bias "ever experimentally recorded" (exceeding 1933 Jim Crow levels) and this problem has gotten worse with larger/more recent models. [bsky, 2 points, 0 comments]
Related