Leveraging In-Context Learning for Political Bias Testing of LLMs

2025/06/27 by Patrick Haller, Jannis Vamvas, Haller, Patrick +5
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Leverage (statistics) #Natural Language Processing Techniques #Political stability #Politics #Stability (learning theory) #Survey data collection #Topic Modeling #Work (physics)

paper · pdf · doi:10.48550/arxiv.2506.22232

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

A growing body of work has been querying LLMs with political questions to evaluate their potential biases. However, this probing method has limited stability, making comparisons between models unreliable. In this paper, we argue that LLMs need more context. We propose a new probing task, Questionnaire Modeling (QM), that uses human survey data as in-context examples. We show that QM improves the stability of question-based bias evaluation, and demonstrate that it may be used to compare instruction-tuned models to their base versions. Experiments with LLMs of various sizes indicate that instruction tuning can indeed change the direction of bias. Furthermore, we observe a trend that larger models are able to leverage in-context examples more effectively, and generally exhibit smaller bias scores in QM. Data and code are publicly available.

Citations

• Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
• Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs
• OLMo: Accelerating the Science of Language Models
• You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments
• More human than human: measuring ChatGPT political bias
• Evaluating the Moral Beliefs Encoded in LLMs
• Mitigating Label Biases for In-context Learning
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• Language (Technology) is Power: A Critical Survey of "Bias" in NLP
• Language Models are Few-Shot Learners
• Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
• Reducing Gender Bias in Abusive Language Detection

Related

• Measuring Political Bias in Large Language Models: What Is Said and How It Is Said
• Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models
• A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models
• Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases
• Adaptive Generation of Bias-Eliciting Questions for LLMs