2022/02/10 by Ikbeom Jang, Garrison Danley, Jang, Ikbeom +5 · 2 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Biological sciences #FOS: Computer and information sciences #FOS: Electrical engineering #I.2.1 #Image and Video Processing (eess.IV) #Machine Learning (cs.LG) #Quantitative Methods (q-bio.QM) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2202.04823
openalex publication_date 2022/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric, demonstrating the effectiveness of the approach in ranking medical images by image quality in this proof of concept study. Using the medical image annotation software that we developed, we actively subsample pairwise comparisons using a sorting algorithm with a human rater in the loop. We find that this method substantially reduces the number of comparisons required for a full ordinal ranking without compromising inter-rater reliability when compared to pairwise comparisons without sorting.