How to train your stochastic parrot: large language models for political texts

2025/01/14 by Joseph T. Ornstein, Elise Blasingame, Jake S. Truscott · 1 voice · 41 citations
Computer Science · Social Sciences · #Computational and Text Analysis Methods #Computer science #Law #Linguistics #Media Influence and Politics #Philosophy #Political science #Politics #Sociology #Topic Modeling

paper · pdf · doi:10.1017/psrm.2024.64

published in Political Science Research and Methods 13(2), 264-281 (Cambridge University Press)

openalex publication_date 2025/01/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Abstract We demonstrate how few-shot prompts to large language models (LLMs) can be effectively applied to a wide range of text-as-data tasks in political science—including sentiment analysis, document scaling, and topic modeling. In a series of pre-registered analyses, this approach outperforms conventional supervised learning methods without the need for extensive data pre-processing or large sets of labeled training data. Performance is comparable to expert and crowd-coding methods at a fraction of the cost. We propose a set of best practices for adapting these models to social science measurement tasks, and develop an open-source software package for researchers.

Citations

• Liberals and conservatives rely on different sets of moral foundations.
• On the Dangers of Stochastic Parrots

Cited by

• Contextualizing Misinformation: A User-Centric Approach to Linguistic and Topical Patterns in News Consumption
• Measuring Scalar Constructs in Social Science with LLMs
• Hostility on Twitter in the aftermath of terror attacks
• Mapping (A)Ideology: A Taxonomy of European Parties Using Generative LLMs as Zero-Shot Learners
• Positioning Political Texts with Large Language Models by Asking and Averaging
• Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
• Generative Large Language Models (gLLMs) in Content Analysis: A Practical Guide for Communication Research
• MarxistLLM: Fine-tuning a language model with a Marxist worldview
• Populism Meets AI: Advancing Populism Research with LLMs
• Nationally Representative, Locally Misaligned: The Biases of Generative Artificial Intelligence in Neighborhood Perception
• Scaling Open-ended Survey Responses Using LLM-Paired Comparisons
• Why Biden-era clean energy investment policies had limited political returns
• Probabilistic Record Linkage Using Pretrained Text Embeddings
• ConflLlama: Domain-specific adaptation of large language models for conflict event classification
• The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks

Discussions

• 🦜New Pub Alert! 🦜 In our new article @psrm.bsky.social, @joeornstein.bsky.social @jaketruscott.bsky.social and I demonstrate how LLMs (like ChaptGPT) can be used to process large text-as-data projec [bsky, 23 points, 1 comments]

Related

• Towards Coding Social Science Datasets with Language Models
• Political-LLM: Large Language Models in Political Science
• Leveraging Large Language Models for Public Opinion and Political Psychology Research: Opportunities and Challenges
• Large Language Models
• Positioning Political Texts with Large Language Models by Asking and Averaging