Increasing Transparency Through a Multiverse Analysis
2016/09/01 by Sara Steegen, Francis Tuerlinckx, Andrew Gelman +1 · 1 voice · 58 citations
Economics, Econometrics and Finance · #Economic and Environmental Valuation
paper · pdf · doi:10.1177/1745691616658637
openalex publication_date 2016/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
Abstract
Empirical research inevitably includes constructing a data set by processing raw data into a form ready for statistical analysis. Data processing often involves choices among several reasonable options for excluding, transforming, and coding data. We suggest that instead of performing only one analysis, researchers could perform a multiverse analysis, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios. Using an example focusing on the effect of fertility on religiosity and political attitudes, we show that analyzing a single data set can be misleading and propose a multiverse analysis as an alternative practice. A multiverse analysis offers an idea of how much the conclusions change because of arbitrary choices in data construction and gives pointers as to which choices are most consequential in the fragility of the result.
Citations
Cited by
- Physiological and communicative emotional disconcordance in children on the autism spectrum
- Open science, the replication crisis, and environmental public health
- Introducing openESM: A database of openly available experience sampling datasets
- Heartbeat-evoked responses in M/EEG: A systematic review of methods with suggestions for analysis and reporting
- Beyond beta rhythms: subthalamic aperiodic broadband power scales with Parkinson's disease severity–a cross-sectional multicentre study
- Making the exercise experience more pleasurable: Exploring the psychological effect of allowing intensity adjustments during exercise
- Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data
- Variability in the analysis of a single neuroimaging dataset by many teams
- The Effect of Different Construct Operationalizations, Study Duration, and Preprocessing Choices on Power-Based Sample Size Recommendations in Intensive Longitudinal Research
- MIO: Multiverse Debugging in the Face of Input/Output -- Extended Version with Additional Appendices
- Enhancing precision in human neuroscience
- How to do Better N400 Studies: Reproducibility, Consistency and Adherence to Research Standards in the Existing Literature
- The JASP guidelines for conducting and reporting a Bayesian analysis
- Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
- The Sci-Hub effect on papers’ citations
- Assessing the validity evidence for habit measures based on time pressure
- Unconscious semantic processing: Insights from metaphorical priming
- A multiverse lack of replication for working memory capacity as moderator of the perceptual disfluency effect
- Quit playing games with our lives: Layoffs predict road traffic fatalities
- Investing in open and FAIR practices for more usable and equitable climate-risk research
- Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
- Sex differences in trust and trustworthiness: A meta-analysis of the trust game and the gift-exchange game
- Cardiovascular recovery from acute psychological stress: Testing the temporal stability of multiple models of recovery
- Individual differences in relational body image: Within-person variability predicts maladaptive trait body image
- Early Vocabulary Acquisition: From Birth Order Effect to Child-to-Caregiver Ratio
- Birth‐order effects on risk taking are limited to the family environment
- RobustiPy: An efficient next generation multiversal library with model selection, averaging, resampling, and explainable artificial intelligence
- Clinical prediction models and the multiverse of madness
- Generalizability of choice architecture interventions
- Abductive Theory Construction
- Introducing multiverse analysis to bibliometrics: The case of team size effects on disruptive research
- We don't know what you did last summer. On the importance of transparent reporting of reaction time data pre-processing
- Protecting against researcher bias in secondary data analysis: challenges and potential solutions
- Same model, same data, but different outcomes: Evaluating the impact of method choices in structural equation modeling
- The replication crisis has led to positive structural, procedural, and community changes
- Predicting replication outcomes and generalizability of results
- On the reliability of value-modulated attentional capture: An online replication and multiverse analysis
- An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)
- Causal Inferences from Digital Behavioral Data
- Toward a more credible assessment of the credibility of science by many-analyst studies
- Whose Party Is This? Explaining Perceptions of US Party Ideology
- A Traveler’s Guide to the Multiverse: Promises, Pitfalls, and a Framework for the Evaluation of Analytic Decisions
- Addressing the theory crisis in psychology
- Reproducible research practices and transparency across linguistics
- Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
- Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
- Misstatements, misperceptions, and mistakes in controlling for covariates in observational research
- Pretty unequal? Immigrant-native differences in returns to physical attractiveness in Germany
- Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
- Do Cryptocurrency Markets Differentiate Infrastructure from Regulatory Shocks? A Multi-Moment Event Study with Dependence-Robust Inference
- The Sci-hub Effect: Sci-hub downloads lead to more article citations
- How Do I Know What My Theory Predicts?
- Does Intergroup Contact Affect Political Attitudes and Behaviours?—A Longitudinal Test of Tertiary Transfer Effects Using the Chilean Longitudinal Social Survey ( ELSOC )
- The case for formal methodology in scientific reform
- The perceptual saliency of fearful eyes and smiles: A signal detection study
- Selective referencing and questionable evidence in Strumia’s paper on “Gender issues in fundamental physics”
- No Support for Historical Candidate Gene or Candidate Gene-by-Interaction Hypotheses for Major Depression Across Multiple Large Samples
- Boba: Authoring and Visualizing Multiverse Analyses
Discussions
Related