Structured Reasoning for Fairness: A Multi-Agent Approach to Bias Detection in Textual Data

2025/03/01 by Huang, Tianyi, Fan, Elsa · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2503.00355

Abstract

From disinformation spread by AI chatbots to AI recommendations that inadvertently reinforce stereotypes, textual bias poses a significant challenge to the trustworthiness of large language models (LLMs). In this paper, we propose a multi-agent framework that systematically identifies biases by disentangling each statement as fact or opinion, assigning a bias intensity score, and providing concise, factual justifications. Evaluated on 1,500 samples from the WikiNPOV dataset, the framework achieves 84.9% accuracy\unicodex2014an improvement of 13.0% over the zero-shot baseline\unicodex2014demonstrating the efficacy of explicitly modeling fact versus opinion prior to quantifying bias intensity. By combining enhanced detection accuracy with interpretable explanations, this approach sets a foundation for promoting fairness and accountability in modern language models.

Cited by

• CEA-LIST at CheckThat! 2025: Evaluating LLMs as Detectors of Bias and Opinion in Text
• The Wisdom of Agent Crowds: A Human-AI Interaction Innovation Ignition Framework

Related

• Fairness and Bias in Multimodal AI: A Survey
• Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases
• Bias and Fairness in Large Language Models: A Survey
• BEATS: Bias Evaluation and Assessment Test Suite for Large Language Models
• Bias and Fairness in Large Language Models: A Survey