AI generates covertly racist decisions about people based on their dialect
2024/08/28 by Valentin Hofmann, Pratyusha Kalluri, Dan Jurafsky +1 · 2 voices · 192 citations
Computer Science · Social Sciences · #Artificial Intelligence in Law #Computer science #Linguistics #Natural Language Processing Techniques #Natural language processing #Philosophy #Ranging #Telecommunications #Topic Modeling
paper · pdf · doi:10.1038/s41586-024-07856-5
published in Nature 633(8028), 147-154 (Nature Portfolio)
openalex publication_date 2024/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
Abstract
Abstract Hundreds of millions of people now interact with language models, with uses ranging from help with writing 1,2 to informing hiring decisions 3 . However, these language models are known to perpetuate systematic racial prejudices, making their judgements biased in problematic ways about groups such as African Americans 4–7 . Although previous research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time, particularly in the United States after the civil rights movement 8,9 . It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice, exhibiting raciolinguistic stereotypes about speakers of African American English (AAE) that are more negative than any human stereotypes about African Americans ever experimentally recorded. By contrast, the language models’ overt stereotypes about African Americans are more positive. Dialect prejudice has the potential for harmful consequences: language models are more likely to suggest that speakers of AAE be assigned less-prestigious jobs, be convicted of crimes and be sentenced to death. Finally, we show that current practices of alleviating racial bias in language models, such as human preference alignment, exacerbate the discrepancy between covert and overt stereotypes, by superficially obscuring the racism that language models maintain on a deeper level. Our findings have far-reaching implications for the fair and safe use of language technology.
Citations
Cited by
- Large Language Models Are Biased Because They Are Large Language Models
- Multimodal large language models can make context-sensitive hate speech evaluations aligned with human judgement
- Training large language models on narrow tasks can lead to broad misalignment
- A Framework for Facilitating Young People’s Sociocritical AI Literacies
- Socially prescriptive speech technologies: Linguistic, technical, and ethical issues
- AI as a talent management tool: An organizational justice perspective
- Ethical Considerations in the Deployment of Artificial Intelligence in Surgery
- AI-Powered Browsers Are Broadly Accurate News Summarizers That Reduce Political Bias and Negative Affect
- Language Models Embody and Amplify Human Cognitive Distortions: What Is to Be Done?
- An approach to systemic risks of AI through the lens of emergence, collective action problems, and externalities
- Data and trained models for "Empirical Evidence of Large Language Model's Influence on Human Spoken Communication"
- Algorithmic Monocultures in Hiring
- A conceptual framework for ideology beyond the left and right
- Large Language Models Reproduce Racial Stereotypes When Used for Text Annotation
- The Digital Divide in Generative AI: Evidence from Large Language Model Use in College Admissions Essays
- Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
- Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems
- A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
- Rejected Dialects: Biases Against African American Language in Reward Models
- Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models
- Biased AI writing assistants shift users’ attitudes on societal issues
- Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
- Responsible Intelligence in Practice: A Fairness Audit of Open Large Language Models for Library Reference Services
- Impacts of Racial Bias in Historical Training Data for News AI
- Linear socio-demographic representations emerge in Large Language Models from indirect cues
- Generative AI Practices, Literacy, and Divides: An Empirical Analysis in the Italian Context
- Us-vs-Them bias in Large Language Models
- Understanding Down Syndrome Stereotypes in LLM-Based Personas
- Writing in Symbiosis: Mapping Human Creative Agency in the AI Era
- Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses
- Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
- AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys
- Designing Beyond Language: Sociotechnical Barriers in AI Health Technologies for Limited English Proficiency
- Large Language Models Develop Novel Social Biases Through Adaptive Exploration
- Why Public Service AI Governance Frameworks Risk Failing in the Age of General-Purpose AI: Lessons from Policing
- Exploring the Intersection of AI, Language, and Law: A Bibliometric Analysis
- Large language models, social demography, and hegemony: comparing authorship in human and synthetic text
- Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks
- DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English
- End-of-life education: An explorative study using artificial intelligence simulations in undergraduate nursing
- Public Opinion on the Politics of AI Alignment: Cross-National Evidence on Expectations for AI Moderation From Germany and the United States
- Identity-Aware Large Language Models require Cultural Reasoning
- Using natural language processing to analyse text data in behavioural science
- The Social Cost of Intelligence: Emergence, Propagation, and Amplification of Stereotypical Bias in Multi-Agent Systems
- Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions
- Assessing Human Rights Risks in AI: A Framework for Model Evaluation
- Disclosure and Evaluation as Fairness Interventions for General-Purpose AI
- Mitigating Biases in Language Models via Bias Unlearning
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Responsible AI in Marketing: AI Booing and AI Washing Cycle of AI Mistrust
- Linguistic Affordances Framework: A Linguistic-Sociological Approach for the Social Study of Language Technology
- Generative AI and linguistic diversity in academic writing and publishing
- AI systems and the reproduction of (standard) language ideologies in World Englishes
- AI as moral cover: How algorithmic bias exploits psychological mechanisms to perpetuate social inequality
- Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset
- Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases
- Designing Effective AI Explanations for Misinformation Detection: A Comparative Study of Content, Social, and Combined Explanations
- AI reasoning effort predicts human decision time in content moderation
- Ask ChatGPT: Caveats and Mitigations for Individual Users of AI Chatbots
- AI-AI Bias: large language models favor communications generated by large language models
- Bias Association Discovery Framework for Open-Ended LLM Generations
- EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Norms
- Language Models Change Facts Based on the Way You Talk
- FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
- Fairness Evaluation of Large Language Models in Academic Library Reference Services
- Exploring LLMs for Automated Pre-Testing of Cross-Cultural Surveys
- Penalizing Transparency? How AI Disclosure and Author Demographics Shape Human and AI Judgments About Writing
- Cash or Comfort? How LLMs Value Your Inconvenience
- Adultification Bias in LLMs and Text-to-Image Models
- Affective-CARA: A Knowledge Graph Driven Framework for Culturally Adaptive Emotional Intelligence in HCI
- AI Agent Behavioral Science
- HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America
- Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
- Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties
- Analyzing values about gendered language reform in LLMs' revisions
- We Need to Measure Data Diversity in NLP -- Better and Broader
- A Survey on Stereotype Detection in Natural Language Processing
- Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4
- Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory
- Read, Critique, or Sketch? Investigating Alternative Visualization Literacy Assessment Modalities
- Not Like Us, Hunty: Measuring Perceptions and Behavioral Effects of Minoritized Anthropomorphic Cues in LLMs
- Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors
- Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
- Emergence of Biased Consensus in Multi-Agent LLM Debates
- From Seeing it to Experiencing it: Interactive Evaluation of Intersectional Voice Bias in Human-AI Speech Interaction
- Artificial intelligence (AI) in social work education and research: reflections from emerging African social work researchers
- Do Language Models Pass the Bechdel Test? Auditing Gender Biases in LLM-Generated Screenplays
- What Do People Actually Want From AI? Mapping Preference Plurality
- Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions
- "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most
- Equitable System-Prompt Selection via Constrained Mixed-Strategy GroupDRO
- From Promising Capability to Pervasive Bias: Assessing Large Language Models for Emergency Department Triage
- Generative language models exhibit social identity biases. [europepmc]
- Modelling the impact of environmental and social determinants on mental health using generative agents. [europepmc]
- Current applications and challenges in large language models for patient care: a systematic review. [europepmc]
- The sociolinguistic foundations of language modeling. [europepmc]
- Gender and racial bias issues in a commercial "tone of voice" analysis system. [europepmc]
- Factors modulating perception and production of speech by AI tools: a test case of Amazon Alexa and Polly. [europepmc]
- Public Perceptions of Judges' Use of AI Tools in Courtroom Decision-Making: An Examination of Legitimacy, Fairness, Trust, and Procedural Justice. [europepmc]
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias. [europepmc]
- Computational challenges arising in algorithmic fairness and health equity with generative AI. [europepmc]
- Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models. [europepmc]
- Examining Chat GPT with nonwords and machine psycholinguistic techniques. [europepmc]
- Embodied artificial intelligence in ophthalmology. [europepmc]
- Biased echoes: Large language models reinforce investment biases and increase portfolio risks of private investors. [europepmc]
- School-Based Online Surveillance of Youth: Systematic Search and Content Analysis of Surveillance Company Websites. [europepmc]
- AI-AI bias: Large language models favor communications generated by large language models. [europepmc]
- Artificial Intelligence in migrant health: a critical perspective on opportunities and risks. [europepmc]
- AI biases as asymmetries: a review to guide practice. [europepmc]
- Training large language models on narrow tasks can lead to broad misalignment. [europepmc]
- Acceptability of Using Large Language Models to Support Smoking Cessation Attempts: a Qualitative Study Among African Americans Who Smoke. [europepmc]
- Evaluation of cross-ethnic emotion recognition capabilities in multimodal large language models using the reading the mind in the eyes test. [europepmc]
- Biased AI writing assistants shift users' attitudes on societal issues. [europepmc]
- Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI. [europepmc]
- Child and adolescent psychiatry: challenges, solutions, opportunities, and future directions. [europepmc]
- Algorithmic bias [wikipedia]
Discussions
- This is the study she’s referencing. Hofmann, V., Kalluri, P.R., Jurafsky, D. et al. (2024) AI generates covertly racist decisions about people based on their dialect. Nature. doi.org/10.1038/s415... [bsky, 31 points, 1 comments]
- Also, the affirmation bot encodes the strongest covert anti-Black bias "ever experimentally recorded" (exceeding 1933 Jim Crow levels) and this problem has gotten worse with larger/more recent models. [bsky, 2 points, 0 comments]
Related