Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodology

2024/06/20 by Federico Ruggeri, Eleonora Misino, Ruggeri, Federico +9 · 2 citations
Biochemistry, Genetics and Molecular Biology · Medicine · #Biomedical Text Mining and Ontologies #Clinical practice guidelines implementation #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2406.14099

openalex publication_date 2024/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce the Guideline-Centered Annotation Methodology (GCAM), a novel data annotation methodology designed to report the annotation guidelines associated with each data sample. Our approach addresses three key limitations of the standard prescriptive annotation methodology by reducing the information loss during annotation and ensuring adherence to guidelines. Furthermore, GCAM enables the efficient reuse of annotated data across multiple tasks. We evaluate GCAM in two ways: (i) through a human annotation study and (ii) an experimental evaluation with several machine learning models. Our results highlight the advantages of GCAM from multiple perspectives, demonstrating its potential to improve annotation quality and error analysis.

Cited by

• Data Annotation Quality Problems in AI-Enabled Perception System Development
• What Has Been Lost with Synthetic Evaluation?

Related

• Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model
• Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
• Synthesized Annotation Guidelines are Knowledge-Lite Boosters for Clinical Information Extraction
• Improving Task Instructions for Data Annotators: How Clear Rules and Higher Pay Increase Performance in Data Annotation in the AI Economy
• Analyzing Dataset Annotation Quality Management in the Wild