2025/10/25 by Sareh Sahebalam, Sahebalam, Sareh, Purya Baghaei +3
Decision Sciences · Social Sciences · #Psychometric Methodologies and Testing #Student Assessment and Feedback #Reliability and Agreement in Measurement
paper · doi:10.57647/jntell.2026.0501.03
Rater-mediated assessments require raters to make complex judgments, often expressed through ordinal rating scale categories, about test-takers’ performances. When the focus is on the quality of these judgments, it becomes essential to evaluate the ratings for their psychometric soundness, including validity, reliability, and fairness. In applied linguistics and second-language assessment research, the Many-Facet Rasch Model (MFRM; Linacre, 1989) has been widely used to investigate rater performance. However, the MFRM’s complexity, combined with the limited accessibility of user-friendly software, presents challenges for researchers and practitioners. In this study, the researchers propose using the Rating Scale Model (RSM), a more straightforward and widely recognized framework, as an alternative for analyzing judged performance. To explore its utility, the RSM was applied to scores assigned by five raters to 156 compositions written by English as a Foreign Language Learners (EFL). The findings indicate that the RSM can successfully convert ordinal ratings into interval-level measures for both test-takers and tasks, while also allowing examination of item fit, person fit, rating scale functioning, and dimensionality. Moreover, the model proved capable of investigating interaction effects, such as the influence of rater and examinee gender, as well as the interplay between raters and specific items. Overall, this study demonstrates that the RSM, while more limited than the MFRM, offers a practical and accessible tool for evaluating the quality of assessor judgments in performance assessment. We also highlight both the promise and the constraints of relying on RSM in this context, with implications for future inquiry and implementation.