Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
2013/01/01 by Justin Grimmer, Brandon M. Stewart, Brandon Stewart · 1 voice · 94 citations
Social Sciences · #Computational and Text Analysis Methods #Electoral Systems and Political Participation #Qualitative Comparative Analysis Research
paper · pdf · doi:10.1093/pan/mps028
openalex publication_date 2013/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
Abstract
Politics and political conflict often occur in the written and spoken word. Scholars have long recognized this, but the massive costs of analyzing even moderately sized collections of texts have hindered their use in political science research. Here lies the promise of automated text analysis: it substantially reduces the costs of analyzing large collections of text. We provide a guide to this exciting new area of research and show how, in many instances, the methods have already obtained part of their promise. But there are pitfalls to using automated methods—they are no substitute for careful thought and close reading and require extensive and problem-specific validation. We survey a wide range of new methods, provide guidance on how to validate the output of the models, and clarify misconceptions and errors in the literature. To conclude, we argue that for automated text methods to become a standard tool for political scientists, methodologists must contribute new methods and new methods of validation.
Citations
Cited by
- Potential and Pitfalls of Audio as Data for Political Research: Alignment, Features, and Classification Models
- Issue Attention in Public Opinion Polls: Pollsters as Agenda Responders and Agenda Setters
- Stay Tuned: Improving Sentiment Analysis and Stance Detection Using Large Language Models
- The use of LLMs to annotate data in management research: Foundational guidelines and warnings
- Generative Large Language Models (gLLMs) in Content Analysis: A Practical Guide for Communication Research
- Support for (liberal) democracy among authoritarian, populist, and radical Western Europeans
- Dirty talk: Media discourse and the struggle over South Africa’s coal transition
- Large Language Models in Thematic Analysis: Prompt Engineering, Evaluation, and Guidelines for Qualitative Software Engineering Research
- MarxistLLM: Fine-tuning a language model with a Marxist worldview
- Latin-American cyborg methods: hacking digital methodologies in and for the scientific south
- Scrolling Through Chaos: The Implications of TikTok for Crisis Sensemaking
- A Hierarchical Error Framework for Reliable Automated Coding in Communication Research: Applications to Health and Political Communication
- MoVa: Towards Generalizable Classification of Human Morals and Values
- Energized? How German and Italian Party Actors Used Emotions in Energy Policy Narratives During the 2024 EU Elections
- The price of automated case law annotation: comparing the cost and performance of GPT-4o and student annotators
- Human-AI Narrative Synthesis to Foster Shared Understanding in Civic Decision-Making
- AWARE-FX: An Auditable Knowledge-Guided AI System for Measuring Corporate Foreign-Exchange Hedging Disclosure
- An Instrument to Evaluate Governance Proposals: AI Policy Analysis at Scale
- Constructing enemies, consolidating rule: Conspiracy narratives and anti-western rhetoric in Erdoğan’s populist discourse
- Rethinking Scaling Up Content Analysis: A Reappraisal of Justifications and Practices for Large-Scale Content Analysis
- Legislators’ sentiment analysis supervised by legislators
- AI in a Comparative Perspective: Linking Policy Formulation and Implementation
- Topics as Outcomes: Modeling the Influence of Intergovernmental Grants on Policy Diffusion
- Probabilistic Record Linkage Using Pretrained Text Embeddings
- Automated Quality Assessment for LLM-Based Complex Qualitative Coding: A Confidence-Diversity Framework
- Online Anti-sexist Speech: Identifying Resistance to Gender Bias in Political Discourse
- What does the public want their local government to hear? A data-driven case study of public comments across the state of Michigan
- Bridging frameworks: A revised approach for evaluating the governance of marine protected areas
- Re‐Imagining the Epistemic Possibilities of GPT for Public Administration Research in Competitive Settings
- Transforming Sensitive Documents into Quantitative Data: An AI-Based Preprocessing Toolchain for Structured and Privacy-Conscious Analysis
- Can a voluntarist approach to social security extend protection to gig workers? Evidence from the platform-based food-delivery sector in China
- Disentangling Similarity and Relatedness in Topic Models
- Digitalization of public sector organizations over time: The applicability of quantitative text analysis
- Agenda Seeding: How 1960s Black Protests Moved Elites, Public Opinion and Voting
- Disintegration and party competition: evidence from parliamentary speeches on Brexit
- The Increased Effect of Elections and Changing Prime Ministers on Topics Discussed in the Australian Federal Parliament between 1901 and 2018
- An Expert-Sourced Measure of Judicial Ideology
- Wordkrill: Extending Wordfish into the multidimensional political space
- What is a high-quality research environment? Evidence from the UK’s research excellence framework
- Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media
- Advancing Harmful Content Detection in Organizational Research: Integrating Large Language Models with Elo Rating System
- Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI
- International organizations in national parliamentary debates
- How to Make Causal Inferences Using Texts
- Machine learning and behavioral economics for personalized choice architecture
- A Two-Sample Test of Text Generation Similarity
- A Novel, Human-in-the-Loop Computational Grounded Theory Framework for Big Social Data
- Machine learning in agricultural and applied economics
- Democratic progress and women’s issue representation: evidence from the South Korean National Assembly, 1948–2022
- How do elites view ideology? Analysing candidates’ associations of left and right
- To Love or to Loathe: How is the World Reacting to China's Rise?
- ‘Time to Wake Up’: Climate change advocacy in a polarized Congress, 1996-2015
- A moral or class divide for populist parties? ‘The people’ in the discourse of Podemos and Vox in Spain
- Anti-environmentalism as conservative coalition maintenance: an automated text analysis of National Review
- Discursive Landscapes and Unsupervised Topic Modeling in IR: A Validation of Text-As-Data Approaches through a New Corpus of UN Security Council Speeches on Afghanistan
- Debating Threats in the UNSC: A New Dataset on Explanations of Votes 1989–2019
- ECONOMETRICS MEETS SENTIMENT: AN OVERVIEW OF METHODOLOGY AND APPLICATIONS
- Coping With the Digital Trilemma? Trade‐Offs and Risks in EU Digital Policy
- Gênero, Dinâmicas de Poder Intrapartidárias e Manterrupting no Legislativo
- QUOTUS: The Structure of Political Media Coverage as Revealed by Quoting Patterns
- Relating Latent Class Assignments to External Variables: Standard Errors for Correct Inference
- A computational social science perspective on qualitative data exploration: Using topic models for the descriptive analysis of social media data*
- Environmental literacy and deliberative democracy: a content analysis of written submissions to the Irish Citizens’ Assembly on climate change
- Media Framing Dynamics of the ‘European Refugee Crisis’: A Comparative Topic Modelling Approach
- Computational Social Science for Nonprofit Studies: Developing a Toolbox and Knowledge Base for the Field
- Advancing Automated Content Analysis for a New Era of Media Effects Research: The Key Role of Transfer Learning
- Equity How and Equity for Whom? Incorporating Equity Into Local Government Budgeting Processes
- Dynamic Surveys: Using LLMs to Blend Qualitative Depth,Quantitative Structure, and Collaborative Interaction
- From Text to Network: Constructing a Knowledge Graph of Taiwan-Based China Studies Using Generative AI
- Quantifying Narrative Similarity Across Languages
- How to make causal inferences using texts
- Identification, Interpretability, and Bayesian Word Embeddings
- Artificial Intelligence for Investigative Reporting
- Communicating with Algorithms: A Transfer Entropy Analysis of Emotions-based Escapes from Online Echo Chambers
- Adapting or Freezing? Ideological Reactions of Communist Regimes to a Post-Communist World
- Modeling Media History
- Constructing war in news: Discursive logics of the Pahalgam crisis in the Indian and Pakistani press
- Inviting the Populists to the Party: Populist Appeals in Presidential Primaries
- Measuring relative opinion from location-based social media: A case study of the 2016 U.S. presidential election
- The Shadow Cabinet in Westminster Systems: Modeling Opposition Agenda Setting in the House of Commons, 1832–1915
- Paradigm Wars Revisited: A Cartography of Graduate Research in the Field of Education (1980–2010)
- When populists become popular: comparing Facebook use by the right-wing movement Pegida and German political parties
- POPmine: Tracking Political Opinion on the Web
- PReSS: An Automated Black-Box Framework for Evaluating Political Stance Stability in LLMs
- Representation Learning for Natural Language Processing
- Missing Data Imputation for Supervised Learning
- Sentiment Aggregate Functions for Political Opinion Polling using Microblog Streams
- Harnessing Latent Dirichlet Allocation-based topic modeling to unveil the focus areas in Ayush healthcare in India
- The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks
- Language Models for Materials Discovery and Sustainability: Progress, Challenges, and Opportunities
- Generating Navigable Semantic Maps from Social Sciences Corpora
- Review Text as a Leading Indicator of Displayed Reputation in Platform Rating Systems: Evidence from 34 U.S. Short-Term Rental Markets
- Computational Basis of LLM's Decision Making in Social Simulation
- Transmissions, Decisions, Discourses – A Methodological Framework for Measuring and Comparing Democratic Innovations’ Policy‐Impact(s)
Discussions
Related