A Note on the Interpretation of Weighted Kappa and its Relations to Other Rater Agreement Statistics for Metric Scales

2004/04/01 by Christof Schuster · 14 citations
Decision Sciences · Mathematics · #Reliability and Agreement in Measurement #Statistical Methods in Epidemiology #Meta-analysis and systematic reviews

paper · doi:10.1177/0013164403260197

Abstract

This article presents a formula for weighted kappa in terms of rater means, rater variances, and the rater covariance that is particularly helpful in emphasizing that weighted kappa is an absolute agreement measure in the sense that it is sensitive to differences in rater’s marginal distributions. Specifically, rater mean differences will decrease the value of weighted kappa relative to the value of the intraclass correlation that ignores mean differences. In addition, if rater variances also differ, then the value of weighted kappa will be decreased relative to the value of the product-moment correlation. Equality constraints on the rater means and variances are given to illustrate the relationships between weighted kappa, the intraclass correlation, and the product-moment correlation. In addition, the expression for weighted kappa shows that weighted kappa belongs to the Zegers-ten Berge family of chance-corrected association coefficients. More specifically, weighted kappa is equivalent to the chance-corrected identity coefficient.

Citations

• A Coefficient of Agreement for Nominal Scales
• The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability

Cited by

• mHealth Series: mHealth project in Zhao County, rural China - Description of objectives, field site and methods. [europepmc]
• The development of the DISCO-RC for measuring children's discomfort during research procedures. [europepmc]
• Validation of the Unesp-Botucatu composite scale to assess acute postoperative abdominal pain in sheep (USAPS). [europepmc]
• Validation of the Donkey Pain Scale (DOPS) for Assessing Postoperative Pain in Donkeys. [europepmc]
• A critical evaluation for validation of composite and unidimensional postoperative pain scales in horses. [europepmc]
• Visual Evaluation of Image Quality of a Low Dose 2D/3D Slot Scanner Imaging System Compared to Two Conventional Digital Radiography X-ray Imaging Systems. [europepmc]
• Analysis of the Weighted Kappa and Its Maximum with Markov Moves. [europepmc]
• Multilingual validation of the short form of the Unesp-Botucatu Feline Pain Scale (UFEPS-SF). [europepmc]
• Validation of the rabbit pain behaviour scale (RPBS) to assess acute postoperative pain in rabbits (Oryctolagus cuniculus). [europepmc]
• The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement. [europepmc]
• Psychometric Properties of the Taiwanese Pressure Ulcer Management Self-Efficacy Scale in Nursing Practice. [europepmc]
• A comprehensive guide to study the agreement and reliability of multi-observer ordinal data. [europepmc]
• Regional climate projections using a deep-learning-based model-ranking and downscaling framework: application to European climate zones. [europepmc]
• Rating scales to inform balance exercise difficulty during rehabilitation for individuals with neurological disorders. [europepmc]

Related

• Coefficient Kappa: Some Uses, Misuses, and Alternatives
• Beyond kappa: A review of interrater agreement measures
• A Proposed Solution to the Base Rate Problem in the Kappa Statistic
• A Linear Relationship between Correlation and Cohen's Kappa for Binary Data and Simulating Multivariate Nominal and Ordinal Data with Specified Kappa Matrix
• Computing inter‐rater reliability and its variance in the presence of high agreement