Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
2023/05/29 by Myra Cheng, Esin Durmus, Cheng, Myra +3 · 1 voice · 66 citations
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer science #Computers and Society (cs.CY) #FOS: Computer and information sciences #Gender studies #Geography #Lexicon #Linguistics #Natural (archaeology) #Persona #Persona Design and Applications #Prejudice (legal term) #Psychology #Social psychology #Sociology
paper · pdf · doi:10.48550/arxiv.2305.18189
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Abstract
To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling. Grounded in the sociolinguistic concept of markedness (which characterizes explicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate personas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written portrayals using the same prompts. The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics. An intersectional lens further reveals tropes that dominate portrayals of marginalized groups, such as tropicalism and the hypersexualization of minoritized women. These representational harms have concerning implications for downstream applications like story generation.
Cited by
- Persona Prompting as a Lens on LLM Social Reasoning
- One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization
- Understanding Down Syndrome Stereotypes in LLM-Based Personas
- Prompt Fairness: Sub-group Disparities in LLMs
- German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies
- Annotating Dimensions of Social Perception in Text: A Sentence-Level Dataset of Warmth and Competence
- Prioritize Economy or Climate Action? Investigating ChatGPT Response Differences Based on Inferred Political Orientation
- Erasing 'Ugly' from the Internet: Propagation of the Beauty Myth in Text-Image Models
- Characterizing Selective Refusal Bias in Large Language Models
- What if AI systems weren't chatbots?
- Ideology-Based LLMs for Content Moderation
- Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback
- Missing the Margins: A Systematic Literature Review on the Demographic Representativeness of LLMs
- Who's Asking? Evaluating LLM Robustness to Inquiry Personas in Factual Question Answering
- Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification
- SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation
- Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy
- Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
- From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education
- Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning
- Speaking with the Past: Constructing AI-Generated Historical Characters for Cultural Heritage and Learning
- Simulating Identity, Propagating Bias: Abstraction and Stereotypes in LLM-Generated Text
- Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases
- Confident, Calibrated, or Complicit: Probing the Trade-offs between Safety Alignment and Ideological Bias in Language Models in Detecting Hate Speech
- "She was useful, but a bit too optimistic": Augmenting Design with Interactive Virtual Personas
- Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs
- DAIQ: Auditing Demographic Attribute Inference from Question in LLMs
- Using AI for User Representation: An Analysis of 83 Persona Prompts
- A Close Reading Approach to Gender Narrative Biases in AI-Generated Stories
- Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models
- The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models
- Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents
- Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models
- Constella: Supporting Storywriters' Interconnected Character Creation through LLM-based Multi-Agents
- Improving the Distributional Alignment of LLMs using Supervision
- Against 'softmaxing' culture
- Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation
- Co-persona: Leveraging LLMs and Expert Collaboration to Understand User Personas through Social Media Data Analysis
- A Hybrid Multi-Agent Prompting Approach for Simplifying Complex Sentences
- Algorithm‐facilitated discrimination: a socio‐legal study of the use by employers of artificial intelligence hiring systems
- Understanding Gender Bias in AI-Generated Product Descriptions
- From Anger to Joy: How Nationality Personas Shape Emotion Attribution in Large Language Models
- Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs
- Localizing Persona Representations in LLMs
- A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contexts
- MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
- Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4
- PersonaBOT: Bringing Customer Personas to Life with LLMs and RAG
- Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization
- Multilingual Prompting for Improving LLM Generation Diversity
- Unraveling Interwoven Roles of Large Language Models in Authorship Privacy: Obfuscation, Mimicking, and Verification
- Language Models That Walk the Talk: A Framework for Formal Fairness Certificates
- From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modeling
- A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias
- Not Like Us, Hunty: Measuring Perceptions and Behavioral Effects of Minoritized Anthropomorphic Cues in LLMs
- Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
- Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions
- "Are we writing an advice column for Spock here?" Understanding Stereotypes in AI Advice for Autistic Users
- Improving Language Model Personas via Rationalization with Psychological Scaffolds
- What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patterns
- Aspirational Affordances of AI
- Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages
- Plausible Patients, Impossible Populations: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations
- Masculine Defaults via Gendered Discourse in Podcasts and Large Language Models
- Societal Impacts Research Requires Benchmarks for Creative Composition Tasks
- How Is Generative AI Used for Persona Development?: A Systematic Review of 52 Research Articles
Discussions
Related