From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models

2025/05/07 by Oscar Stuhler, Cat Dang Ton, Étienne Ollion · 1 voice · 16 citations
Computer Science · Social Sciences · #Advanced Text Analysis Techniques #Artificial intelligence #Computational and Text Analysis Methods #Computer science #Generative grammar #Generative model #Linguistics #Natural language processing #Topic Modeling

paper · pdf · doi:10.1177/00491241251336794

published in Sociological Methods & Research 54(3), 794-848 (SAGE Publishing)

openalex publication_date 2025/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into “promptbooks.” We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE while also underscoring risks like non-random error patterns that can bias downstream analyses.

Citations

• What do animals do all day?: The division of labor, class bodies, and totemic thinking in the popular imagination
• ChatGPT outperforms crowd workers for text-annotation tasks

Cited by

• Computational Text Analysis for Building and Testing Social Theory
• Critical, but constructive: defining, detecting, and addressing bias in Computational Social Science
• Measuring complex constructs in large-scale text with computational social mixed methods
• Efficiency vs. understanding: a critical examination of ChatGPT’s performance in context-sensitive annotation tasks
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• Time and Climate Change: U.S. Media Representations of Climate Actions, Horizons, and Events (2000 to 2021)
• Linguistic Affordances Framework: A Linguistic-Sociological Approach for the Social Study of Language Technology
• Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy
• RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models
• Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale
• An Introduction to Generative Artificial Intelligence for Academics. [europepmc]

Discussions

• NEW ARTICLE: Want to use LLMs to extract information at scale in sociology? @eollion.bsky.social @oms279.bsky.social and C. Ton have you covered. Read "From Codebooks to Promptbooks," now in Sociologi [bsky, 49 points, 1 comments]

Related

• From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models
• Large Language Models for Generative Information Extraction: A Survey
• Big Meaning: Qualitative Analysis on Large Bodies of Data Using AI
• GPT models for text annotation: An empirical exploration in public policy research
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets