2026/04/01 by David P. Wilkinson, Fallon Mody, Mark A. Burgman +27 · 1 voice
Decision Sciences · Psychology · Social Sciences · #Meta-analysis and systematic reviews #Mental Health Research Topics #Computational and Text Analysis Methods
paper · doi:10.31222/osf.io/hj9ex_v1
openalex publication_date 2026/04/01 · openalex created_date 2026/04/02 · openalex updated_date 2026/07/15
The repliCATS-SCORE dataset consists of five human elicitation studies predicting the replicability of published research in the social and behavioural sciences, conducted by the repliCATS project (Collaborative Assessments for Trustworthy Science) as part of DARPA’s Systematizing Confidence in Open Research and Evidence or SCORE program (2019-2022). The human elicitations were conducted using a structured group deliberation technique – the IDEA protocol – for a total of 4000 published research claims and an additional 25 claim pilot study. The dataset for each study includes participant-level judgements across two elicitation rounds, mathematically aggregated group judgements or “confidence scores” calculated using a suite of 38 aggregation methods, as well as participant free-text reasoning. The repliCATS-SCORE dataset can be re-used in support of studies interrogating scholarly judgements of replicability, plausibility and overall credibility, post-publication peer review and critical evaluation training, developing novel mathematical aggregation models that forecast replication outcomes, and comparative studies.