2024/07/02 by Christopher Barrie, Barrie, Christopher, Elli Palaiologou +3 · 2 voices · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2407.02039
openalex publication_date 2024/07/02 · arxiv published 2024/07/02 · openalex created_date 2025/10/10 · arxiv updated 2026/05/15 · openalex updated_date 2026/07/28
Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package promptstability for its estimation. Using six different datasets and twelve outcomes, we classify ∼3.1m rows of data and ∼300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.