Positioning Political Texts with Large Language Models by Asking and Averaging

2023/11/28 by Gaël Le Mens, Aina Gallego, Mens, Gaël Le +1 · 2 voices · 17 citations
Computer Science · Mathematics · Social Sciences · #Correlation #Crowdsourcing #Econometrics #Economics #Electoral Systems and Political Participation #Ideology #Law #Mathematics #Natural Language Processing Techniques #Political science #Politics #Position (finance) #Scaling #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2311.16639

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/11/28 · openalex created_date 2023/11/30 · openalex updated_date 2026/08/01

Abstract

We use instruction-tuned Large Language Models (LLMs) like GPT-4, Llama 3, MiXtral, or Aya to position political texts within policy and ideological spaces. We ask an LLM where a tweet or a sentence of a political text stands on the focal dimension and take the average of the LLM responses to position political actors such as US Senators, or longer texts such as UK party manifestos or EU policy speeches given in 10 different languages. The correlations between the position estimates obtained with the best LLMs and benchmarks based on text coding by experts, crowdworkers, or roll call votes exceed .90. This approach is generally more accurate than the positions obtained with supervised classifiers trained on large amounts of research data. Using instruction-tuned LLMs to position texts in policy and ideological spaces is fast, cost-efficient, reliable, and reproducible (in the case of open LLMs) even if the texts are short and written in different languages. We conclude with cautionary notes about the need for empirical validation.

Cited by

• Validating LLMs in social science: Epistemic threats and emerging norms
• The Netherlands Local Manifesto Project. Infrastructure for studying local party politics
• Computational Measurement of Political Positions: A Review of Text-Based Ideal Point Estimation Algorithms
• Combining LLM Semantic Reasoning with GNN Structural Modeling for Multi-View Multi-Label Feature Selection
• Who Drives the Security Narrative in US Trade Policy?
• Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
• MarxistLLM: Fine-tuning a language model with a Marxist worldview
• AI and Human Oversight: A Risk-Based Framework for Alignment
• News under pressure? Politicization of scientific expertise in German mainstream news during COVID-19
• LLM Analysis of 150+ years of German Parliamentary Debates on Migration Reveals Shift from Post-War Solidarity to Anti-Solidarity in the Last Decade
• Learning the Topic, Not the Language: How LLMs Classify Online Immigration Discourse Across Languages
• Elite Polarization in European Parliamentary Speeches: a Novel Measurement Approach Using Large Language Models
• Stay Tuned: Improving Sentiment Analysis and Stance Detection Using Large Language Models
• Wordkrill: Extending Wordfish into the multidimensional political space
• Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media
• Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approaches
• Mapping (A)Ideology: A Taxonomy of European Parties Using Generative LLMs as Zero-Shot Learners

Discussions

• Hey, first post here :) Gael Le Mens and I just uploaded a paper about scaling political text using ChatGPT. This approach is fast, cost-efficient, and reliable. We'd love to hear your thoughts and f [bsky, 68 points, 0 comments]
• Oh boy. This is either awesome or awful (didn't have time to read the paper yet; see here: arxiv.org/abs/2311.16639 ) Is it time to finally retire DW-NOMINATE? (spoiler: I don't think so) #Polisky #S [bsky, 2 points, 0 comments]

Related

• Positioning Political Texts with Large Language Models by Asking and Averaging
• Large language models as a substitute for human experts in annotating political text
• Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages
• Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers
• Enhancing LLM-Based Text Classification in Political Science: Automatic Prompt Optimization and Dynamic Exemplar Selection for Few-Shot Learning