Structural Topic Models for Open‐Ended Survey Responses

2014/03/06 by Margaret E. Roberts, Brandon M. Stewart, Brandon Stewart +6 · 1,966 citations
Psychology · Social Sciences · #Coding (social sciences) #Computational and Text Analysis Methods #Computer science #Data science #Information retrieval #Political science #Politics #Psychology #Social science #Sociology

paper · doi:10.1111/ajps.12103

published in American Journal of Political Science 58(4), 1064-1082 (Wiley)

openalex publication_date 2014/03/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Collection and especially analysis of open‐ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semiautomated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author's gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open‐ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.

Citations

• Probabilistic topic models

Cited by

• What Good Is a Regression? Inference to the Best Explanation and the Practice of Political Science Research
• How do voters interpret social class appeals? Lessons from open-ended responses
• Computational Text Analysis for Building and Testing Social Theory
• International bureaucrats under transparency: The case of the WTO TRIPS Council
• Contextualizing Misinformation: A User-Centric Approach to Linguistic and Topical Patterns in News Consumption
• Far-right agenda setting: How the far right influences the political mainstream
• Beyond standardization: a comprehensive review of topic modeling validation methods for computational social science research
• Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale
• Representation increases women's influence in climate deliberations: Evidence from community‐managed forests in Malawi
• Seeded Topic Models in Digital Archives: Analyzing Interpretations of Immigration in Swedish Newspapers, 1945–2019
• Transmissions, Decisions, Discourses – A Methodological Framework for Measuring and Comparing Democratic Innovations’ Policy‐Impact(s)
• What's a parent to do? Measuring cultural logics of parenting with computational text analysis
• Evaluating the causal mechanisms from procedural justice to police legitimacy and compliance in the United States: An in‐depth mixed‐methods approach
• Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys
• Degrowth and postgrowth: A systematic literature review of growth-critical science
• Analyzing Political Text at Scale with Online Tensor LDA
• LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives
• Reppin’ your constituency? Urban-rural differences in geographical representation in Swedish parliamentary debates
• Digital Prompting in Education: A Design Framework and Bibliometric Analysis
• Multidimensional conflicts over disarmament and international security: analyzing speeches in the First Committee of the UN General Assembly
• Measuring Political Attitudes with Word Association
• Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
• Spelling correction with large language models to reduce measurement error in open-ended survey responses
• From Symbols to Embeddings: A Tale of Two Representations in Computational Social Science
• The sound of respondents: predicting respondents’ level of interest in questions with voice data in smartphone surveys
• Rethinking Reputation: When Fighting to Demonstrate Resolve Backfires
• Effects of Embodied Interviewing Agents on Open Narrative Responses
• Through Thick and Thin: Comparing Traditional Qualitative Analysis and Natural Language Processing Techniques Using Narrative Data from Police Officers
• Does Outsourcing Reduce Public Scrutiny Toward Elected Officials?: Insights From US Municipal Water Management
• Wayfarers in Cyberspace: A Temporal Investigation of Digital Nomads Based on Liquid Modernity Theory
• Natural Language Processing in Psychology
• Beliefs about Climate Beliefs: The Importance of Second-Order Opinions for Climate Politics
• Election Campaigning on Social Media: Politicians, Audiences, and the Mediation of Political Communication on Facebook and Twitter
• Why are carbon taxes unfair? Disentangling public perceptions of fairness
• Text as Behavior
• Strategy Framing in News Coverage and Electoral Success: An Analysis of Topic Model Networks Approach
• Time and Climate Change: U.S. Media Representations of Climate Actions, Horizons, and Events (2000 to 2021)
• Farmers’ Voices in European Protests: Diverse Complaints, Emotional Tones, and Policy Responses
• Support for (liberal) democracy among authoritarian, populist, and radical Western Europeans
• Stress and coping in VRChat: A mixed-method case study of the use of VRChat as a coping tool during the COVID-19 pandemic
• Asking the Right Questions: A Framework for Developing Gender-Balanced Political Knowledge Batteries
• Using artificial intelligence to support self-regulated learning: A systematic review of empirical research
• Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs
• The limits of AI for authoritarian control
• Theory-Driven Analysis of Large Corpora: Semisupervised Topic Classification of the UN Speeches
• Scaling Open-ended Survey Responses Using LLM-Paired Comparisons
• Text Mining of Open-Ended Questions in Self-Assessment of University Teachers: An LDA Topic Modeling Approach
• The evolution of discourses on livestock farming in Germany: a text mining analysis of parliamentary debates (1949–2021)
• SNAP judgments into the digital age: Reporting on food stamps varies significantly with time, publication type, and political leaning
• Self-coding: A method to assess semantic validity and bias when coding open-ended responses
• Ideology from topic mixture statistics: inference method and example application to carbon tax public opinion
• The Development of an Issue Public: Evidence from The Eras Tour
• Evolution of research on global amphibian declines
• Application of Natural Language Processing to Determine User Satisfaction in Public Services
• Topic Modeling and Text Analysis for Qualitative Policy Research
• Crash Themes in Automated Vehicles: A Topic Modeling Analysis of the California Department of Motor Vehicles Automated Vehicle Crash Database
• Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy
• Hamiltonian Monte Carlo for Regression with High-Dimensional Categorical\n Data
• Diverse Effects of Mass Media on Concerns about Immigration: New Evidence from Germany, 2001–2016
• Mixed Membership Word Embeddings for Computational Social Science
• Leveraging Media Frames to Improve Normative Diversity in News Recommendations
• Exploring links between national climate strategies and non-state and subnational climate action in nationally determined contributions (NDCs)
• Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
• Text Mining for Processing Interview Data in Computational Social Science
• Objectifying the Subjective: Cognitive Biases in Topic Interpretations
• Computer-Assisted Text Analysis for Social Science: Topic Models and Beyond
• Do LGBTIQ+ issues matter for populist radical right? An analysis of Italian parties’ social media narratives
• Inviting the Populists to the Party: Populist Appeals in Presidential Primaries
• You are right. I am ALARMED -- But by Climate Change Counter Movement
• Detecting latent topics and trends in educational technologies over four decades using structural topic modeling: A retrospective of all volumes of Computers & Education
• Exploring the Early Adoption of Open AI among Laypeople and Technical Professionals: An Analysis of Twitter Conversations on #ChatGPT and #GPT3
• Quantifying Narrative Similarity Across Languages
• From tragedy to advocacy: how celebrity suicides transform mental health conversations on Facebook
• A Computational Social Science Approach to Understanding Predictors of Chafee Service Receipt
• Citizens at the forefront of the constitutional debate: Participation determinants and emergent content in Chile
• Large Language Models for Statistical Inference: Context Augmentation with Applications to the Two-Sample Problem and Regression
• Generalizability of Media Frames: Corpus creation and analysis across countries
• Detecting Narrative Shifts through Persistent Structures: A Topological Analysis of Media Discourse
• Mental Illness and the Resilience of Deviance
• The Political Psychology of Cancel Culture: Value Framing or Group Identity?
• Mapping soft skills and further research directions for higher education: a bibliometric approach with structural topic modelling
• PersonaBOT: Bringing Customer Personas to Life with LLMs and RAG
• Topic-based classification and identification of global trends for startup companies
• A Machine-Learning Analysis of the Impacts of the COVID-19 Pandemic on Small Business Owners and Implications for Canadian Government Policy Response
• The Diversity-Innovation Paradox in Science
• Political Common Ground on Preserving Nature: Environmental Motives Across the Political Spectrum
• Purifying the Leviathan: The Strategic Dilemma of an Anti-Corruption Campaign Under One-Party Rule
• Innovating Web Probing: Comparing Written and Oral Answers to Open-Ended Probing Questions in a Smartphone Survey
• When Disagreement Becomes Uncivil on Social Media: The Role of Passive Receiving and Active Expression of Incivility in Influencing Political Polarization
• How to make causal inferences using texts
• Automatic Claim Review for Climate Science via Explanation Generation
• The Anchoring Effect in Decision-Making with Visual Analytics
• Adapting or Freezing? Ideological Reactions of Communist Regimes to a Post-Communist World
• Beyond Fact-Checking: Network Analysis Tools for Monitoring Disinformation in Social Media
• Paradigm Wars Revisited: A Cartography of Graduate Research in the Field of Education (1980–2010)
• Actors, Frames and Arguments: A Multi-Decade Computational Analysis of Climate Discourse in Financial News using Large Language Models
• Corporate funding and ideological polarization about climate change
• An Image is Worth K Topics: A Visual Structural Topic Model with Pretrained Image Embeddings
• Democratic classification of free-format survey responses with a network-based framework

Related

• Computer-Assisted Text Analysis for Social Science: Topic Models and Beyond
• Topic Modeling for Free-Response Text Data from a Complex Survey
• Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation
• Keyword Assisted Topic Models
• Topic Modeling and Text Analysis for Qualitative Policy Research