Min-Mid-Max Scaling, Limits of Agreement, and Agreement Score

2020/06/23 by Veli Safak, Safak, Veli · 1 citation
Decision Sciences · Mathematics · #FOS: Computer and information sciences #Machine Learning (stat.ML) #Methodology (stat.ME) #Multi-Criteria Decision Making #Reliability and Agreement in Measurement #Statistical Methods in Epidemiology #stat.ME #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.12904

Cohen's kappa, limits of agreement

openalex publication_date 2020/06/23 · arxiv created 2021/03/13 · arxiv updated 2021/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, I solve a 60-year old question posed by Cohen's seminal paper (1960) and offer an agreement measure centered around the chance-expected agreement while isolating marginally forced agreement and disagreement. To achieve this, I formulate the minimum feasible agreement given row and column marginals by devising a new algorithm that minimizes the sum of diagonals in contingency tables. Based on this result, I also formulate the lower limit of the most common agreement measure-Cohen's kappa. Finally, I study the lower limit of maximum feasible agreement and devise two statistics of distribution similarity for agreement analysis.

Citations

• High agreement but low Kappa: I. the problems of two paradoxes
• A Coefficient of Agreement for Nominal Scales
• The Measurement of Observer Agreement for Categorical Data
• The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements

Cited by

• Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers

Related

• Explicit agreement extremes for a 2×2 table with given marginals
• Better to be in agreement than in bad company: a critical analysis of many kappa-like tests assessing one-million 2x2 contingency tables
• Significance Tests for the Measure of Raw Agreement
• Measuring intergroup agreement and disagreement
• Agreement Implies Accuracy for Substitutable Signals