2025/06/24 by Sadia Kamal, Lalu Prasad Yadav Prakash, Kamal, Sadia +9 · 1 citation
Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Media Influence and Politics #Populism, Right-Wing Movements
paper · pdf · doi:10.48550/arxiv.2506.22493
openalex publication_date 2025/06/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.