Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of 'Bias', and Academia-Industry Gap
2025/08/14 by Ghosh, Sourojit, Wilson, Kyra
#Computers and Society (cs.CY) #FOS: Computer and information sciences
paper · doi:10.48550/arxiv.2508.11067
Abstract
The rapid development of AI tools and implementation of LLMs within downstream tasks has been paralleled by a surge in research exploring how the outputs of such AI/LLM systems embed biases, a research topic which was already being extensively explored before the era of ChatGPT. Given the high volume of research around the biases within the outputs of AI systems and LLMs, it is imperative to conduct systematic literature reviews to document throughlines within such research. In this paper, we conduct such a review of research covering AI/LLM bias in four premier venues/organizations -- *ACL, FAccT, NeurIPS, and AAAI -- published over the past 10 years. Through a coverage of 189 papers, we uncover patterns of bias research and along what axes of human identity they commonly focus. The first emergent pattern within the corpus was that 82% (155/189) papers did not establish a working definition of "bias" for their purposes, opting instead to simply state that biases and stereotypes exist that can have harmful downstream effects while establishing only mathematical and technical definition of bias. 94 of these 155 papers have been published in the past 5 years, after Blodgett et al. (2020)'s literature review with a similar finding about NLP research and recommendation to consider how such researchers should conceptualize bias, going beyond strictly technical definitions. Furthermore, we find that a large majority of papers -- 79.9% or 151/189 papers -- focus on gender bias (mostly, gender and occupation bias) within the outputs of AI systems and LLMs. By demonstrating a strong focus within the field on gender, race/ethnicity (30.2%; 57/189), age (20.6%; 39/189), religion (19.1%; 36/189) and nationality (13.2%; 25/189) bias, we document how researchers adopt a fairly narrow conception of AI bias by overlooking several non-Western communities in fairness research, as we advocate for a stronger coverage of such populations. Finally, we note that while our corpus contains several examples of innovative debiasing methods across the aforementioned aspects of human identity, only 10.6% (20/189) include recommendations for how to implement their findings or contributions in real-world AI systems or design processes. This indicates a concerning academia-industry gap, especially since many of the biases that our corpus contains several successful mitigation methods that still persist within the outputs of AI systems and LLMs commonly used today. We conclude with recommendations towards future AI/LLM fairness research, with stronger focus on diverse marginalized populations.
Citations
- Documenting Patterns of Exoticism of Marginalized Populations within Text-to-Image Generators
- Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework
- BendVLM: Test-Time Debiasing of Vision-Language Embeddings
- Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
- A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
- Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI
- "I don't see myself represented here at all": User Experiences of Stable Diffusion Outputs Containing Representational Harms across Gender Identities and Nationalities
- Interpretations, Representations, and Stereotypes of Caste within Text-to-Image Generators
- How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies
- BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs
- Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictions
- FairJob: A Real-World Dataset for Fairness in Online Systems
- Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Models
- MBIAS: Mitigating Bias in Large Language Models While Retaining Context
- Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
- PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models
- FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations
- Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation
- Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception
- MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressions
- Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency
- Humans or LLMs as the Judge? A Study on Judgement Biases
- CultureLLM: Incorporating Cultural Differences into Large Language Models
- IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators
- Whose wife is it anyway? Assessing bias against same-gender relationships in machine translation
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Stable Diffusion Exposed: Gender Bias from Prompt to Image
- Fast Model Debias with Machine Unlearning
- "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
- Building Socio-culturally Inclusive Stereotype Resources with Community Engagement
- VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
- Sociodemographic Bias in Language Models: A Survey and Forward Path
- T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation
- Uncovering and Quantifying Social Biases in Code Generation
- Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
- Comparing Biases and the Impact of Multilingual Training across Multiple Languages
- Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- Stable Bias: Analyzing Societal Representations in Diffusion Models
- Logic Against Bias: Textual Entailment Mitigates Stereotypical Sentence Reasoning
- Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?
- Conceptor-Aided Debiasing of Large Language Models
- Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
- MABEL: Attenuating Gender Bias using Textual Entailment Data
- Controlling Bias Exposure for Fair Interpretable Predictions
- Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving
- Balancing out Bias: Achieving Fairness Through Balanced Training
- On Measures of Biases and Harms in NLP
- The Values Encoded in Machine Learning Research
- Quantifying Social Biases in NLP: A Generalization and Empirical\n Comparison of Extrinsic Fairness Metrics
- Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
- Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
- A large-scale study on research code quality and execution
- Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
- Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- Language (Technology) is Power: A Critical Survey of "Bias" in NLP
- Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem
- Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
- Closing the AI accountability gap
- Assessing Social and Intersectional Biases in Contextualized Word Representations
- Literature review as a research methodology: An overview and guidelines
- On Measuring and Mitigating Biased Inferences of Word Embeddings
- Model Cards for Model Reporting
- Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
- Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
- Bias in computer systems
Related