Updating “The Future of Coding”: Qualitative Coding with Generative Large Language Models

2025/05/21 by Nga Than, Leanne Fan, Tina Law +2 · 40 citations
Computer Science · Social Sciences · #Artificial intelligence #Coding (social sciences) #Computational and Text Analysis Methods #Computer science #Generative grammar #Linguistics #Natural Language Processing Techniques #Natural language processing #Qualitative research #Social science #Sociology #Topic Modeling

paper · doi:10.1177/00491241251339188

published in Sociological Methods & Research 54(3), 849-888 (SAGE Publishing)

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/02

Abstract

Over the past decade, social scientists have adapted computational methods for qualitative text analysis, with the hope that they can match the accuracy and reliability of hand coding. The emergence of GPT and open-source generative large language models (LLMs) has transformed this process by shifting from programming to engaging with models using natural language, potentially mimicking the in-depth, inductive, and/or iterative process of qualitative analysis. We test the ability of generative LLMs to replicate and augment traditional qualitative coding, experimenting with multiple prompt structures across four closed- and open-source generative LLMs and proposing a workflow for conducting qualitative coding with generative LLMs. We find that LLMs can perform nearly as well as prior supervised machine learning models in accurately matching hand-coding output. Moreover, using generative LLMs as a natural language interlocutor closely replicates traditional qualitative methods, indicating their potential to transform the qualitative research process, despite ongoing challenges.

Citations

• Computational Grounded Theory: A Methodological Framework
• The Future of Coding: A Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods
• Flexible Coding of In-depth Interviews: A Twenty-first-century Approach
• On the Dangers of Stochastic Parrots
• ChatGPT outperforms crowd workers for text-annotation tasks
• Machine Bias. How Do Generative Language Models Answer Opinion Polls? <sup/>
• Large Language Models for Text Classification: From Zero-Shot Learning to Instruction-Tuning
• On the Opportunities and Risks of Foundation Models
• Language Models are Few-Shot Learners

Cited by

• Integrating Generative Artificial Intelligence into Social Science Research: Measurement, Prompting, and Simulation
• Computational Text Analysis for Building and Testing Social Theory
• Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• When LLMs fall short in Deductive Coding: Model Comparison and Human AI Collaboration Workflow Design
• Humanlike AI Design Increases Anthropomorphism but Yields Divergent Outcomes on Engagement and Trust Globally
• Integrating Computational Methods and AI into Qualitative Studies of Aging and Later Life
• Generative AI in Sociological Research: State of the Discipline
• The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
• Studying Culture and Meaning Through Interpretative Computational Methods: From theory to method and back
• Introducing HALC: A general pipeline for finding optimal prompting strategies for automated coding with LLMs in the computational social sciences
• Multi-Agent Ethnography: Post-Conventional Anthropological Practice Through Human−AI Collaboration

Related

• When Qualitative Research Meets Large Language Model: Exploring the Potential of QualiGPT as a Tool for Qualitative Coding
• QualiGPT: GPT as an easy-to-use tool for qualitative coding
• Large Language Model for Qualitative Research -- A Systematic Mapping Study
• Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks
• Prompts Matter: Comparing ML/GAI Approaches for Generating Inductive Qualitative Coding Results