2023/02/28 by Katie Wolfe, Michael A. Seaman · 1 citation
Decision Sciences · Psychology · #Reliability and Agreement in Measurement #Safety Warnings and Signage
paper · pdf · doi:10.1002/jaba.980
Visual analysis is the primary method of analyzing single-case research data, yet relatively little is known about the variables that influence raters' decisions and rater agreement. Previous research has suggested that trend, variability, and autocorrelation may negatively affect interrater agreement, but studies have been limited by small numbers of graphs and participants whose knowledge of single-case research was not described. The purpose of this study was to examine the main and interaction effects of two values of each of six data characteristics (e.g., level, trend, and number of data points) on agreement among visual analysts. Using data from Lanovaz and Hranchuk (2021), we examined odds ratios to identify data characteristics that influence interrater agreement. Results suggest that trend and effect size, and to a lesser extent variability, have the largest effects on interrater agreement. We discuss the implications of our results for future research on improving interrater agreement among visual analysts.