How much does outlier and influence handling change meta-analytic conclusions?

2026/07/02 by Tomáš Havránek, Zuzana Iršová, T. D. Stanley +1 · 1 voice

paper · doi:10.17605/osf.io/97cmv

openalex publication_date 2026/07/02 · openalex created_date 2026/07/03 · openalex updated_date 2026/07/13

Abstract

This registration is the pre-analysis plan for a descriptive study of how much routine outlier and influence handling changes meta-analytic conclusions. Across a large, comparable corpus of published meta-analyses in psychology, psychotherapy, and exercise research (all on the Cohen's d scale, drawn from BEAR), five common handling rules are applied under two estimators (random effects with the Hartung–Knapp adjustment, and unrestricted weighted least squares) and we record how far the pooled effect moves, whether its statistical significance changes, and whether it still reaches a smallest effect size of interest. The full plan, the frozen input dataset (with its SHA-256 checksum and the exact BEAR source commit), and the complete analysis code are attached. The plan is registered before the analysis is run.

Discussions

• Almost all the flips are in results already sitting on the threshold. Strongly significant findings essentially never change. Pre-registration: doi.org/10.17605/OSF... Replication: doi.org/10.5281/ze [bsky, 0 points, 0 comments]

Related

• Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses
• Fixed‐ versus random‐effects models in meta‐analysis: Model properties and an empirical comparison of differences in results
• What can we learn from 1,000 meta-analyses across 10 different disciplines?
• Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study
• Power analysis for random‐effects meta‐analysis