Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study

2025/10/06 by Feihao Chen, Majumdar, Ayan, Jinghui Li +4 · 1 citation
Computer Science · #Audit #Benchmark (surveying) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Focus (optics) #Machine Learning (cs.LG) #Regulatory focus theory #Scalability #Sentiment Analysis and Opinion Mining #Social media #Strengths and weaknesses #Task (project management)

paper · pdf · doi:10.48550/arxiv.2510.04641

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/06 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Large-scale web-scraped text corpora used to train general-purpose AI models often contain harmful demographic-targeted social biases, creating a regulatory need for data auditing and developing scalable bias-detection methods. Although prior work has investigated biases in text datasets and related detection methods, these studies remain narrow in scope. They typically focus on a single content type (e.g., hate speech), cover limited demographic axes, overlook biases affecting multiple demographics simultaneously, and analyze limited techniques. Consequently, practitioners lack a holistic understanding of the strengths and limitations of recent large language models (LLMs) for automated bias detection. In this study, we conduct a comprehensive benchmark study on English texts to assess the ability of LLMs in detecting demographic-targeted social biases. To align with regulatory requirements, we frame bias detection as a multi-label task of detecting targeted identities using a demographic-focused taxonomy. We then systematically evaluate models across scales and techniques, including prompting, in-context learning, and fine-tuning. Using twelve datasets spanning diverse content types and demographics, our study demonstrates the promise of fine-tuned smaller models for scalable detection. However, our analyses also expose persistent gaps across demographic axes and multi-demographic targeted biases, underscoring the need for more effective and scalable detection frameworks.

Citations

• Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models
• Qwen2.5 Technical Report
• Class-RAG: Real-Time Content Moderation with Retrieval Augmented Generation
• SLM-Mod: Small Language Models Surpass LLMs at Content Moderation
• Efficacy of Synthetic Data as a Benchmark
• ShieldGemma: Generative AI Content Moderation Based on Gemma
• The Llama 3 Herd of Models
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• TrustLLM: Trustworthiness in Large Language Models
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Benchmarking Cognitive Biases in Large Language Models as Evaluators
• Bias and Fairness in Large Language Models: A Survey
• Bias and Fairness in Large Language Models: A Survey
• NBIAS: A Natural Language Processing Framework for Bias Identification in Text
• CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models
• WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• Fairness of ChatGPT
• A Holistic Approach to Undesired Content Detection in the Real World
• Analyzing Hate Speech Data along Racial, Gender and Intersectional Axes
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
• RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
• HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
• Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender\n Bias
• CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Language (Technology) is Power: A Critical Survey of "Bias" in NLP
• Language Models are Few-Shot Learners
• StereoSet: Measuring stereotypical bias in pretrained language models
• Social Bias Frames: Reasoning about Social and Power Implications of Language
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Gender Bias in Coreference Resolution
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
• Equality of Opportunity in Supervised Learning

Related

• Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation
• Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases
• Exploring Subjectivity for more Human-Centric Assessment of Social Biases in Large Language Models
• BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMs
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases