Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models

2024/07/15 by Louis Abraham, Abraham, Louis, Charles Arnal +3 · 1 voice · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computational and Text Analysis Methods #Computers and Society (cs.CY) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.CY

paper · pdf · doi:10.48550/arxiv.2407.10645

openalex publication_date 2024/07/15 · arxiv published 2024/07/15 · arxiv updated 2025/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models have recently been applied to text annotation tasks from social sciences, equalling or surpassing the performance of human workers at a fraction of the cost. However, no inquiry has yet been made on the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .

Citations

• Quantifying and alleviating political bias in language models
• ChatGPT outperforms crowd workers for text-annotation tasks
• The Unreasonable Effectiveness of Eccentric Automatic Prompts

Cited by

• A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages

Discussions

• Happy to have assisted data scientists Louis Abraham and Charles Arnal on this working paper: "Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models" arx [bsky, 3 points, 1 comments]

Related

• Best Practices for Text Annotation with Large Language Models
• Best Practices for Text Annotation with Large Language Models
• Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting
• Large Language Model Prompt Datasets: An In-depth Analysis and Insights
• iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loop