Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and Considerations in Real-World Scenarios

2023/06/26 by NamHyeok Kim, Kim, NamHyeok, Chanjun Park +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2306.14373

openalex publication_date 2023/06/26 · openalex created_date 2023/06/28 · openalex updated_date 2026/07/28

Abstract

Inter-Annotator Agreement (IAA) is commonly used as a measure of label consistency in natural language processing tasks. However, in real-world scenarios, IAA has various roles and implications beyond its traditional usage. In this paper, we not only consider IAA as a measure of consistency but also as a versatile tool that can be effectively utilized in practical applications. Moreover, we discuss various considerations and potential concerns when applying IAA and suggest strategies for effectively navigating these challenges.

Related

• Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
• Transcending Traditional Boundaries: Leveraging Inter-Annotator Agreement (IAA) for Enhancing Data Management Operations (DMOps)
• Inter-Coder Agreement for Computational Linguistics
• Anchoring and Agreement in Syntactic Annotations
• You Are What You Annotate: Towards Better Models through Annotator Representations