2025/01/01 by Tamkinat Rauf, Jan G. Voelkel, Jan Gerrit Voelkel +2 · 4 voices · 3 citations
Mathematics · Social Sciences · #Advanced Causal Inference Techniques #Audit #Causal inference #Inference #Moderation #Null hypothesis #Outcome (game theory) #Qualitative Comparative Analysis Research #Research design #Sample (material) #Sample size determination #Statistical power #Survey Methodology and Nonresponse
paper · doi:10.1093/poq/nfaf052
published in Public Opinion Quarterly 89(4), 1063-1086 (Oxford University Press)
openalex publication_date 2025/01/01 · openalex created_date 2025/11/28 · openalex updated_date 2026/08/06
Abstract Survey experiments have become a popular methodology for causal inference across the social sciences. We study the efficacy of survey experiment designs by analyzing 100 social science experiments—entailing more than 1,000 hypothesis tests—that were selected by experts via a competitive process and fielded on probability samples of US adults between 2012 and 2020. Inclusion in the analysis is only conditional on the experiment qualifying for data collection, and not in any way on study results or publication. Results show that less than a third of proposed hypotheses were supported by the data, implying many more null findings than ostensibly appear in the published literature. We find that the largest predictor of positive experimental results was sample size. This is somewhat surprising, given that experimental studies typically take power considerations into account prior to data collection. In our data, the importance of sample size stemmed from small effect sizes across studies (perhaps smaller than researchers may have anticipated), highlighting a tension between commonly used power calculi and determining what constitutes a “meaningful effect.” We also find that moderation hypotheses were rarely significant, and that using multiple items for outcome measures did not affect results as expected. But indicators of research experience predicted higher rates of positive results, suggesting that there may be some room for optimizing experiment outcomes by minimizing design errors.