Language (Technology) is Power: A Critical Survey of "Bias" in NLP
2020/05/28 by Su Lin Blodgett, Solon Barocas, Blodgett, Su Lin +6 · 2 voices · 181 citations
Computer Science · #cs.CL #cs.CY
paper · pdf · doi:10.48550/arxiv.2005.14050
arxiv created 2020/05/29 · arxiv updated 2020/06/01
Abstract
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing "bias" is an inherently normative process. We further find that these papers' proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. Based on these findings, we describe the beginnings of a path forward by proposing three recommendations that should guide work analyzing "bias" in NLP systems. These recommendations rest on a greater recognition of the relationships between language and social hierarchies, encouraging researchers and practitioners to articulate their conceptualizations of "bias"---i.e., what kinds of system behaviors are harmful, in what ways, to whom, and why, as well as the normative reasoning underlying these statements---and to center work around the lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations between technologists and such communities.
Cited by
- Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
- Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
- Explaining GAND: A Resource on Gender-Ambiguous Natural Data & Contrastive Attribution
- On The Conceptualization and Societal Impact of Cross-Cultural Bias
- Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?
- Identifying Features Associated with Bias Against 93 Stigmatized Groups in Language Models and Guardrail Model Safety Mitigation
- Teaching and Critiquing Conceptualization and Operationalization in NLP
- Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation
- Identifying Bias in Machine-generated Text Detection
- What Triggers my Model? Contrastive Explanations Inform Gender Choices by Translation Models
- LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples
- A Unifying Human-Centered AI Fairness Framework
- AfriStereo: A Culturally Grounded Dataset for Evaluating Stereotypical Bias in Large Language Models
- BERnaT: Basque Encoders for Representing Natural Textual Diversity
- Understanding Down Syndrome Stereotypes in LLM-Based Personas
- Are LLMs Good Safety Agents or a Propaganda Engine?
- Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses
- Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation
- Empathetic Cascading Networks: A Multi-Stage Prompting Technique for Reducing Social Biases in Large Language Models
- Bias in, Bias out: Annotation Bias in Multilingual Large Language Models
- ‘We can see a savage’: a case study of the colonial gaze in generative AI algorithms
- Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback
- Annotating Dimensions of Social Perception in Text: A Sentence-Level Dataset of Warmth and Competence
- From Double to Triple Burden: Gender Stratification in the Latin American Data Annotation Gig Economy
- Large Language Models Develop Novel Social Biases Through Adaptive Exploration
- Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data
- LLMs Do Not See Age: Assessing Demographic Bias in Automated Systematic Review Synthesis
- Advancing Equitable AI: Evaluating Cultural Expressiveness in LLMs for Latin American Contexts
- Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
- Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens
- Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs
- Back to the Communities: A Mixed-Methods and Community-Driven Evaluation of Cultural Sensitivity in Text-to-Image Models
- Exploring the Intersection of AI, Language, and Law: A Bibliometric Analysis
- Biases in the Blind Spot: Detecting What LLMs Fail to Mention
- Political censorship in large language models originating from China
- Ideology-Based LLMs for Content Moderation
- Can ChatGPT Code Communication Data Fairly?: Empirical Evidence from Multiple Collaborative Tasks
- "You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations
- Identity-Aware Large Language Models require Cultural Reasoning
- CRepair Wrapper Improves Structural Self-Repair Across Three LLM Families: A Cross-Model Replication Study
- ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios
- MarxistLLM: Fine-tuning a language model with a Marxist worldview
- Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions
- Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains
- Large Language Models: A Paradigm Shift for Dementia Diagnosis and Care
- Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
- The fragility of "cultural tendencies" in LLMs
- Assessing Human Rights Risks in AI: A Framework for Model Evaluation
- Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens
- Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study
- Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directions
- BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models
- Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaning
- Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?
- Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
- Aligning Recommendations with User Popularity Preferences
- The Sound of Silencing: Identities and Ideologies in Commercial Text-To-Speech
- Gender Bias in English-to-Greek Machine Translation
- DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture
- HICode: Hierarchical Inductive Coding with LLMs
- Intrinsic Meets Extrinsic Fairness: Assessing the Downstream Impact of Bias Mitigation in Large Language Models
- Philosophy-informed Machine Learning
- Bias Amplification in Stable Diffusion's Representation of Stigma Through Skin Tones and Their Homogeneity
- A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
- Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages
- Gender-Neutral Rewriting in Italian: Models, Approaches, and Trade-offs
- PromptGuard: An Orchestrated Prompting Framework for Principled Synthetic Text Generation for Vulnerable Populations using LLMs with Enhanced Safety, Fairness, and Controllability
- Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
- From Detection to Mitigation: Addressing Gender Bias in Chinese Texts via Efficient Tuning and Voting-Based Rebalancing
- Augmenting Human-Centered Racial Covenant Detection and Georeferencing with Plug-and-Play NLP Pipelines
- Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
- El peaje de los de abajo: midiendo el peaje lingüístico (comprensión, calidad y costo) del español colombiano frente a los LLM
- Applications and Challenges of Fairness APIs in Machine Learning Software
- Mitigation of Gender and Ethnicity Bias in AI-Generated Stories through Model Explanations
- Clustering Discourses: Racial Biases in Short Stories about Women Generated by Large Language Models
- Where Should I Study? Biased Language Models Decide! Evaluating Fairness in LMs for Academic Recommendations
- Culture is Everywhere: A Call for Intentionally Cultural Evaluation
- Social Bias in Multilingual Language Models: A Survey
- Bias-Adjusted LLM Agents for Human-Like Decision-Making via Behavioral Economics
- ISCA: A Framework for Interview-Style Conversational Agents
- Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs
- DAIQ: Auditing Demographic Attribute Inference from Question in LLMs
- Speciesism in AI: Evaluating Discrimination Against Animals in Large Language Models
- Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of 'Bias', and Academia-Industry Gap
- BIPOLAR: Polarization-based granular framework for LLM bias evaluation
- Yet another algorithmic bias: A Discursive Analysis of Large Language Models Reinforcing Dominant Discourses on Gender and Race
- PakBBQ: A Culturally Adapted Bias Benchmark for QA
- "Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
- Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution
- Fair Play in the Newsroom: Actor-Based Filtering Gender Discrimination in Text Corpora
- I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations
- Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approach
- Moving beyond harm. A critical review of how NLP research approaches discrimination
- A Survey on Data Security in Large Language Models
- Bias Association Discovery Framework for Open-Ended LLM Generations
- Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English
- Unequal Voices: How LLMs Construct Constrained Queer Narratives
- Factual Inconsistencies in Multilingual Wikipedia Tables
- Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making
- Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language
- GG-BBQ: German Gender Bias Benchmark for Question Answering
- PrefPalette: Personalized Preference Modeling with Latent Attributes
- A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
- EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering
- When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
- Bias-Aware Mislabeling Detection via Decoupled Confident Learning
- FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
- NLP Meets the World: Toward Improving Conversations With the Public About Natural Language Processing Research
- Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generation
- Mathematics Isn't Culture-Free: Probing Cultural Gaps via Entity and Scenario Perturbations
- FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes
- Leveraging In-Context Learning for Political Bias Testing of LLMs
- Public Service Algorithm: towards a transparent, explainable, and scalable content curation for news content based on editorial values
- If Deceptive Patterns are the problem, are Fair Patterns the solution?
- Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications
- Theories of "Sexuality" in Natural Language Processing Bias Research
- Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
- What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunity
- Distinguishing Task-Specific and General-Purpose AI in Regulation
- Gender Inclusivity Fairness Index (GIFI): A Multilevel Framework for Evaluating Gender Diversity in Large Language Models
- SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
- Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
- Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages
- Intersectional Bias in Japanese Large Language Models from a Contextualized Perspective
- Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge
- Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment
- How do datasets, developers, and models affect biases in a low-resourced language?
- Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomes
- SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
- A Heuristic Perspective on Debiasing Language Models
- A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas
- Words of Warmth: Trust and Sociability Norms for over 26k English Words
- EuroGEST: Investigating gender stereotypes in multilingual language models
- Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework
- Understanding Gender Bias in AI-Generated Product Descriptions
- taz2024full: Analysing German Newspapers for Gender Bias and Discrimination across Decades
- From Anger to Joy: How Nationality Personas Shape Emotion Attribution in Large Language Models
- Visualization for interactively adjusting the de-bias effect of word embedding
- Different Speech Translation Models Encode and Translate Speaker Gender Differently
- Whispers of Many Shores: Cultural Alignment through Collaborative Cultural Expertise
- Multiple LLM Agents Debate for Equitable Cultural Alignment
- Framing Political Bias in Multilingual LLMs Across Pakistani Languages
- Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
- MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
- Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
- Analyzing values about gendered language reform in LLMs' revisions
- Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systems
- Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
- Developing A Framework to Support Human Evaluation of Bias in Generated Free Response Text
- We Need to Measure Data Diversity in NLP -- Better and Broader
- Language Models Should be Used to Surface the Unwritten Code of Science and Society
- Social Bias in Popular Question-Answering Benchmarks
- Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory
- What is Stigma Attributed to? A Theory-Grounded, Expert-Annotated Interview Corpus for Demystifying Mental-Health Stigma
- Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds
- Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation
- Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods
- Should AI Mimic People? Understanding AI-Supported Writing Technology Among Black Users
- Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures
- Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias
- Invisible failures in human-AI interactions
- LLM BiasScope: A Real-Time Bias Analysis Platform for Comparative LLM Evaluation
- Do Language Models Pass the Bechdel Test? Auditing Gender Biases in LLM-Generated Screenplays
- Consistency in Language Models: Current Landscape, Challenges, and Future Directions
- AI Evaluation Should Require Standardized Item-Level Data Releases
- Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
- BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models
- Who Gets to Do Physics? Occupational Stereotypes in AI-Generated Problem Sets
- Interactive Discovery and Exploration of Visual Bias in Generative Text-to-Image Models
- Bye Bye Perspective API: Lessons for Measurement Infrastructure in NLP, CSS and LLM Evaluation
- Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs
- "Are we writing an advice column for Spock here?" Understanding Stereotypes in AI Advice for Autistic Users
- Splits! A Flexible Dataset and Evaluation Framework for Sociocultural Linguistic Investigation
- Combating Toxic Language: A Review of LLM-Based Strategies for Software Engineering
- Tackling Social Bias against the Poor: A Dataset and Taxonomy on Aporophobia
- An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation
- Decolonizing Linguistic Policies in Automated Speech Recognition: A Framework for Cross-Culturally Competent Speech AI
- Masculine Defaults via Gendered Discourse in Podcasts and Large Language Models
- Bias Beyond English: Evaluating Social Bias and Debiasing Methods in a Low-Resource Setting
- An Evaluation of Cultural Value Alignment in LLM
- MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered
- Toward Holistic Evaluation of Recommender Systems Powered by Generative Models
Discussions
Related