Structural Topic Models for Open‐Ended Survey Responses

2014/03/06 by Margaret E. Roberts, Brandon M. Stewart, Brandon Stewart +6 · 67 citations
Social Sciences · #Computational and Text Analysis Methods

paper · doi:10.1111/ajps.12103

Abstract

Collection and especially analysis of open‐ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semiautomated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author's gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open‐ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.

Citations

• Probabilistic topic models

Cited by

• What Good Is a Regression? Inference to the Best Explanation and the Practice of Political Science Research
• How do voters interpret social class appeals? Lessons from open-ended responses
• Computational Text Analysis for Building and Testing Social Theory
• International bureaucrats under transparency: The case of the WTO TRIPS Council
• Contextualizing Misinformation: A User-Centric Approach to Linguistic and Topical Patterns in News Consumption
• Far-right agenda setting: How the far right influences the political mainstream
• Beyond standardization: a comprehensive review of topic modeling validation methods for computational social science research
• Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale
• Representation increases women's influence in climate deliberations: Evidence from community‐managed forests in Malawi
• Seeded Topic Models in Digital Archives: Analyzing Interpretations of Immigration in Swedish Newspapers, 1945–2019
• Transmissions, Decisions, Discourses – A Methodological Framework for Measuring and Comparing Democratic Innovations’ Policy‐Impact(s)
• What's a parent to do? Measuring cultural logics of parenting with computational text analysis
• Evaluating the causal mechanisms from procedural justice to police legitimacy and compliance in the United States: An in‐depth mixed‐methods approach
• Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys
• Degrowth and postgrowth: A systematic literature review of growth-critical science
• Analyzing Political Text at Scale with Online Tensor LDA
• LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives
• Reppin’ your constituency? Urban-rural differences in geographical representation in Swedish parliamentary debates
• Digital Prompting in Education: A Design Framework and Bibliometric Analysis
• Multidimensional conflicts over disarmament and international security: analyzing speeches in the First Committee of the UN General Assembly
• Measuring Political Attitudes with Word Association
• Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
• Spelling correction with large language models to reduce measurement error in open-ended survey responses
• From Symbols to Embeddings: A Tale of Two Representations in Computational Social Science
• The sound of respondents: predicting respondents’ level of interest in questions with voice data in smartphone surveys
• Rethinking Reputation: When Fighting to Demonstrate Resolve Backfires
• Effects of Embodied Interviewing Agents on Open Narrative Responses
• Through Thick and Thin: Comparing Traditional Qualitative Analysis and Natural Language Processing Techniques Using Narrative Data from Police Officers
• Does Outsourcing Reduce Public Scrutiny Toward Elected Officials?: Insights From <scp>US</scp> Municipal Water Management
• Wayfarers in Cyberspace: A Temporal Investigation of Digital Nomads Based on Liquid Modernity Theory
• Natural Language Processing in Psychology
• Beliefs about Climate Beliefs: The Importance of Second-Order Opinions for Climate Politics
• Election Campaigning on Social Media: Politicians, Audiences, and the Mediation of Political Communication on Facebook and Twitter
• Why are carbon taxes unfair? Disentangling public perceptions of fairness
• Text as Behavior
• Strategy Framing in News Coverage and Electoral Success: An Analysis of Topic Model Networks Approach
• Time and Climate Change: U.S. Media Representations of Climate Actions, Horizons, and Events (2000 to 2021)
• Farmers’ Voices in European Protests: Diverse Complaints, Emotional Tones, and Policy Responses
• Support for (liberal) democracy among authoritarian, populist, and radical Western Europeans
• Stress and coping in VRChat: A mixed-method case study of the use of VRChat as a coping tool during the COVID-19 pandemic
• Asking the Right Questions: A Framework for Developing Gender-Balanced Political Knowledge Batteries
• Using artificial intelligence to support self-regulated learning: A systematic review of empirical research
• Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs
• The limits of AI for authoritarian control
• Theory-Driven Analysis of Large Corpora: Semisupervised Topic Classification of the UN Speeches
• Scaling Open-ended Survey Responses Using LLM-Paired Comparisons
• Text Mining of Open-Ended Questions in Self-Assessment of University Teachers: An LDA Topic Modeling Approach
• The evolution of discourses on livestock farming in Germany: a text mining analysis of parliamentary debates (1949–2021)
• SNAP judgments into the digital age: Reporting on food stamps varies significantly with time, publication type, and political leaning
• Self-coding: A method to assess semantic validity and bias when coding open-ended responses
• Ideology from topic mixture statistics: inference method and example application to carbon tax public opinion
• The Development of an Issue Public: Evidence from The Eras Tour
• Evolution of research on global amphibian declines
• Application of Natural Language Processing to Determine User Satisfaction in Public Services
• Topic Modeling and Text Analysis for Qualitative Policy Research
• Crash Themes in Automated Vehicles: A Topic Modeling Analysis of the\n California Department of Motor Vehicles Automated Vehicle Crash Database
• Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy
• Hamiltonian Monte Carlo for Regression with High-Dimensional Categorical\n Data
• Diverse Effects of Mass Media on Concerns about Immigration: New Evidence from Germany, 2001–2016
• Mixed Membership Word Embeddings for Computational Social Science
• Leveraging Media Frames to Improve Normative Diversity in News Recommendations
• Exploring links between national climate strategies and non-state and subnational climate action in nationally determined contributions (NDCs)
• Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
• Objectifying the Subjective: Cognitive Biases in Topic Interpretations
• Do LGBTIQ+ issues matter for populist radical right? An analysis of Italian parties’ social media narratives
• Inviting the Populists to the Party: Populist Appeals in Presidential Primaries
• You are right. I am ALARMED -- But by Climate Change Counter Movement

Related

• Computer-Assisted Text Analysis for Social Science: Topic Models and Beyond
• Topic Modeling for Free-Response Text Data from a Complex Survey
• Creating Targeted, Interpretable Topic Models with LLM-Generated Text Augmentation
• Keyword Assisted Topic Models
• Topic Modeling and Text Analysis for Qualitative Policy Research