A Synthetic Prediction Market for Estimating Confidence in Published Work

2021/12/23 by Sarah Rajtmajer, Rajtmajer, Sarah, Christopher Griffin +27 · 1 citation
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computational and Text Analysis Methods #Computers and Society (cs.CY) #Data Analysis with R #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Multiagent Systems (cs.MA)

paper · pdf · doi:10.48550/arxiv.2201.06924

openalex publication_date 2021/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and behavioral sciences literature. We demonstrate our system and detail our findings using a collection of known replication projects. We suggest that this work lays the foundation for a research agenda that creatively uses AI for peer review.

Cited by

• Examining the replicability of online experiments selected by a decision market

Related

• Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models
• Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
• Peer Prediction for Peer Review: Designing a Marketplace for Ideas
• repliCATS-SCORE: Elicited human predictions of social and behavioural science replicability
• Perspectives from India: Opportunities and Challenges for AI Replication Prediction to Improve Confidence in Published Research