2014/03/31 by Lutz Bornmann, Bornmann, Lutz · 1 citation
Computer Science · Decision Sciences · Mathematics · Physics and Astronomy · #Digital Libraries (cs.DL) #FOS: Computer and information sciences #FOS: Physical sciences #Meta-analysis and systematic reviews #Physics and Society (physics.soc-ph) #Reliability and Agreement in Measurement #Statistical Methods in Clinical Trials #cs.DL #physics.soc-ph
paper · pdf · doi:10.48550/arxiv.1404.0359
Accepted for publication in the Journal of the Association for Information Science and Technology
openalex publication_date 2014/03/31 · arxiv created 2014/06/03 · arxiv updated 2014/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Peer review is the backbone of modern science. F1000Prime is a post-publication peer review system of the biomedical literature (papers from medical and biological journals). This study is concerned with the inter-rater reliability and convergent validity of the peer recommendations formulated in the F1000Prime peer review system. The study is based on around 100,000 papers with recommendations from Faculty members. Even if intersubjectivity plays a fundamental role in science, the analyses of the reliability of the F1000Prime peer review system show a rather low level of agreement between Faculty members. This result is in agreement with most other studies which have been published on the journal peer review system. Logistic regression models are used to investigate the convergent validity of the F1000Prime peer review system. As the results show, the proportion of highly cited papers among those selected by the Faculty members is significantly higher than expected. In addition, better recommendation scores are also connected with better performance of the papers.