Beyond Western Politics: Cross-Cultural Benchmarks for Evaluating Partisan Associations in LLMs

2025/09/24 by Divyanshu Kumar, Kumar, Divyanshu, Ishita Gupta +9
Business, Management and Accounting · #Computers and Society (cs.CY) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Political Influence and Corporate Strategies

paper · pdf · doi:10.48550/arxiv.2509.22711

openalex publication_date 2025/09/24 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Partisan bias in LLMs has been evaluated to assess political leanings, typically through a broad lens and largely in Western contexts. We move beyond identifying general leanings to examine harmful, adversarial representational associations around political leaders and parties. To do so, we create datasets NeutQA-440 (non-adversarial prompts) and AdverQA-440 (adversarial prompts), which probe models for comparative plausibility judgments across the USA and India. Results show high susceptibility to biased partisan associations and pronounced asymmetries (e.g., substantially more favorable associations for U.S. Democrats than Republicans) alongside mixed-polarity concentration around India's BJP, highlighting systemic risks and motivating standardized, cross-cultural evaluation.

Citations

• FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes
• Language-Dependent Political Bias in AI: A Study of ChatGPT and Gemini
• Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models
• Artificial Barriers to Intelligence - Training Effects on Response Patterns in Large Language Models
• Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models
• A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions
• Eliminating Position Bias of Language Models: A Mechanistic Approach
• Assessing Political Bias in Large Language Models
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models
• Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction
• Bias and Fairness in Large Language Models: A Survey
• Bias and Fairness in Large Language Models: A Survey
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models

Related

• Beyond Partisan Leaning: A Comparative Analysis of Political Bias in Large Language Models
• Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditor
• Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification
• Through the LLM Looking Glass: A Socratic Probing of Donkeys, Elephants, and Markets
• A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models