2014/12/01 by Jeremy Howick, Alexander Mebius · 1 citation
Decision Sciences · Economics, Econometrics and Finance · Mathematics · Medicine · #Meta-analysis and systematic reviews #Health Systems, Economic Evaluations, Quality of Life #Statistical Methods in Clinical Trials #Randomized controlled trial #Blinding #Randomization #Medicine #Sample size determination #MEDLINE #Treatment effect #Randomized experiment #Statistics #Surgery #Traditional medicine #Pathology
paper · pdf · doi:10.1186/1745-6215-15-480
openalex publication_date 2014/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23
A 2011 Cochrane Review found that adequately randomized trials sometimes revealed larger, sometimes smaller, and often similar effect sizes to inadequately randomized trials. However, they found no average statistically significant difference in effect sizes between the two study types. Yet instead of concluding that adequate randomization had no effect the review authors postulated the "unpredictability paradox", which states that randomized and non-randomized studies differ, but in an unpredictable direction. However, stipulating the unpredictability paradox is problematic for several reasons: 1) it makes the authors' conclusion that adequate randomization makes a difference unfalsifiable—if it turned out that adequately randomized trials had significantly different average results from inadequately randomized trials the authors could have pooled the results and concluded that adequate randomization protected against bias; 2) it leaves other authors of reviews with similar results confused about whether or not to pool results (and hence which conclusions to draw); 3) it discourages researchers from investigating the conditions under which adequate randomization over- or under-exaggerates apparent treatment benefits; and 4) it could obscure the relative importance of allocation concealment and blinding which may be more important than adequate randomization.