Coefficient Kappa: Some Uses, Misuses, and Alternatives
1981/10/01 by Robert L. Brennan, Dale J. Prediger · 1,559 citations
Decision Sciences · Mathematics · #A priori and a posteriori #Categorical variable #Cohen's kappa #Computer science #Data mining #Econometrics #Kappa #Mathematics #Measure (data warehouse) #Multi-Criteria Decision Making #Reliability (semiconductor) #Reliability and Agreement in Measurement #Statistical Methods in Epidemiology #Statistics #Term (time)
paper · pdf · doi:10.1177/001316448104100307
published in Educational and Psychological Measurement 41(3), 687-699 (SAGE Publishing)
openalex publication_date 1981/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/03
Abstract
This paper considers some appropriate and inappropriate uses of coefficient kappa and alternative kappa-like statistics. Discussion is restricted to the descriptive characteristics of these statistics for measuring agreement with categorical data in studies of reliability and validity. Special consideration is given to assumptions about whether marginals are fixed a priori, or free to vary. In reliability studies, when marginals are fixed, coefficient kappa is found to be appropriate. When either or both of the marginals are free to vary, however, it is suggested that the "chance" term in kappa be replaced by 1/ n, where n is the number of categories. In validity studies, we suggest considering whether one wants an index of improvement beyond "chance" or beyond the best a priori strategy employing base rates. In the former case, considerations are similar to those in reliability studies with the marginals for the criterion measure considered as fixed. In the latter case, it is suggested that the largest marginal proportion for the criterion measure be used in place of the "chance" term in kappa. Similarities and differences among these statistics are discussed and illustrated with synthetic data.
Citations
Cited by
- The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements
- Measuring complex constructs in large-scale text with computational social mixed methods
- Social Story Frames: Contextual Reasoning about Narrative Intent and Reception
- Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit*
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks
- The Eigenvalues Entropy as a Classifier Evaluation Measure
- Birthday memories: an experimental think-aloud study on autobiographical remembering in the digital age
- Chance‐Corrected Interrater Agreement Statistics for Two‐Rater Dichotomous Responses: A Method Review With Comparative Assessment Under Possibly Correlated Decisions
- Do Journalists’ Political Orientations Translate into Partisan News Reporting? The Limits of Bias and the Limits of Counter Mechanisms
- Can we use automated approaches to measure the quality of online political discussion? How to (not) measure interactivity, diversity, rationality, and incivility in online comments to the news
- Microplastics in food and drink: perceptions of the risks, challenges, and solutions among individuals in the ‘farm-to-fork’ food chain
- Mediated autobiographical remembering in the digital age: insights from an experimental think-aloud study
- From the shadows into the light: How pretrial publicity and deliberation affect mock jurors’ decisions, impressions, and memory.
- Obesity in the New Media: A Content Analysis of Obesity Videos on YouTube
- Energized? How German and Italian Party Actors Used Emotions in Energy Policy Narratives During the 2024 EU Elections
- Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
- Automated measurement of infant and mother Duchenne facial expressions in the Face‐to‐Face/Still‐Face
- The impact of bionic prostheses on users' self-perceptions: A qualitative study
- Using Computational Grounded Theory to Analyze Pre-service Chemistry Teachers’ Reflective Practice Regarding Technology Integration in Classrooms Within a Service-Learning–Oriented Seminar
- Media consolidation and news content quality
- An Alternative Measure of Effect Size for Cochran's Q Test for Related Proportions
- FM-LC: A Hierarchical Framework for Urban Flood Mapping by Land Cover Identification Models
- Significance Tests for the Measure of Raw Agreement
- Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability
- Attitudes toward milk alternatives and motives for (non-)consumption: an interview study with adolescents from Germany
- Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoning
- Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
- Exploring folk theories of algorithmic news curation for explainable design
- Identifying Explanation Needs: Towards a Catalog of User-based Indicators
- Evaluation of utilisation and consequences of CRP point-of-care-testing in primary care practices: qualitative interviews with GPs from Germany
- Scientific peer review
- Lehramtsstudierende im Master analysieren inklusiven Naturwissenschaftsunterricht – die Entwicklung von Analysekompetenzen mittels videostimulierter Reflexionen
- Quantum-Driven Multihead Inland Waterbody Detection With Transformer-Encoded CYGNSS Delay-Doppler Map Data
- Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications
- On hallucinations and memory: the relationship between hallucinations and autobiographical overgenerality in Alzheimer’s Disease
- Reliability and Expected Loss: A Unifying Principle
- Relative expertise in an everyday reasoning task: Epistemic understanding, problem representation, and reasoning competence
- An Examination of Interrater Reliability for Scoring the Rorschach Comprehensive System in Eight Data Sets
- When effect sizes disagree: The case of r and d.
- Gender, Race, and Risk: Intersectional Risk Management in the Sale of Sex Online
- Participation of Children With Developmental Language Disorders in Educational Settings—Parents' Perspectives on Patterns, Environmental Influences and Support Strategies
- Hyperspectral Calibration Detection: A Novel Concept For Change Detection With Unsupervised Incremental Safe Pseudo-Labeling Implementation
- When nationalism beats populism: The secessionist movement in Texas
- Classification and treatment of proximal humerus fractures: inter-observer reliability and agreement across imaging modalities and experience. [europepmc]
- Developing a standardized approach to the assessment of pain in children and youth presenting to pediatric rheumatology providers: a Delphi survey and consensus conference process followed by feasibility testing. [europepmc]
- Molecular characterisation of cell line models for triple-negative breast cancers. [europepmc]
- Measuring the quality of patients' goals and action plans: development and validation of a novel tool. [europepmc]
- Experience Corps Baltimore: Exploring the Stressors and Rewards of High-intensity Civic Engagement. [europepmc]
- DWI intensity values predict FLAIR lesions in acute ischemic stroke. [europepmc]
- Observer agreement paradoxes in 2x2 tables: comparison of agreement measures. [europepmc]
- A large-scale analysis of sex differences in facial expressions. [europepmc]
- Patient recall of specific cognitive therapy contents predicts adherence and outcome in adults with major depressive disorder. [europepmc]
- Shear wave elastography and Afirma™ gene expression classifier in thyroid nodules with indeterminate cytology: a comparison study. [europepmc]
- Misinformation on vaccination: A quantitative analysis of YouTube videos. [europepmc]
- Emotional and Phenomenological Properties of Odor-Evoked Autobiographical Memories in Alzheimer's Disease. [europepmc]
- Multitask learning and benchmarking with clinical time series data. [europepmc]
- Why Cohen's Kappa should be avoided as performance measure in classification. [europepmc]
- Facebook Groups on Chronic Obstructive Pulmonary Disease: Social Media Content Analysis. [europepmc]
- How Health Communication via Tik Tok Makes a Difference: A Content Analysis of Tik Tok Accounts Run by Chinese Provincial Health Committees. [europepmc]
- Investigating the Acceptance of Video Consultation by Patients in Rural Primary Care: Empirical Comparison of Preusers and Actual Users. [europepmc]
- Influence of a novel deep-learning based reconstruction software on the objective and subjective image quality in low-dose abdominal computed tomography. [europepmc]
- Processing Body Image on Social Media: Gender Differences in Adolescent Boys' and Girls' Agency and Active Coping. [europepmc]
- Cost of Myopia Correction: A Systematic Review. [europepmc]
- Performance of the Digital Dietary Assessment Tool MyFoodRepo. [europepmc]
- Couples Coping With Hematological Cancer: Support Within and Outside the Couple - Findings From a Qualitative Analysis of Dyadic Interviews. [europepmc]
- Interrater reliability estimators tested against true interrater reliabilities. [europepmc]
- Developing a Healthcare Technology Acceptance Model (H-TAM) for Older Adults with Hypertension. [europepmc]
- Undergraduate Biology Lecture Courses Predominantly Test Facts about Science Rather than Scientific Practices. [europepmc]
Related