Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

2022/10/28 by Nate Breznau, Eike Mark Rinke, Alexander Wuttke +97 · 2 voices · 58 citations
Decision Sciences · #scientometrics and bibliometrics research

paper · doi:10.1073/pnas.2203150119

Abstract

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Citations

• The association between adolescent well-being and digital technology use

Cited by

• To fill or not to fill: Comparing imputation methods for improved riverine long‐term biodiversity monitoring
• The Increase in Refugees to Germany and Exclusionary Beliefs and Behaviors
• Ideological bias in the production of research findings
• Are our 95% CIs only worth 45% confidence? A heterogeneity-aware method for extending inference
• PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework
• Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
• Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel
• The effect of seeing scientists as intellectually humble on trust in scientists and their research
• A Tale of Open Science: Emergence of a New Normal
• Grilling the data: application of specification curve analysis to red meat and all-cause mortality
• Do fiscal rules undermine public investments? A review of empirical evidence
• Is economics self‐correcting? Replications in the <i>American Economic Review</i>
• Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
• The Case for Preregistering Quasi-Experimental Program and Policy Evaluations
• Immigration and Public Support for Social Policy: Accounting for the Gender Composition of Immigrant Populations
• Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis
• Developing synthetic microdata through machine learning for firm-level business surveys
• Investigating the analytical robustness of the social and behavioural sciences
• Causal thinking for decision making on Electronic Health Records: why and how
• Same model, same data, but different outcomes: Evaluating the impact of method choices in structural equation modeling
• Reproducibility and robustness of economics and political science research
• New Data Sources for Demographic Research
• The replication crisis has led to positive structural, procedural, and community changes
• The Agentic Garden of Forking Paths
• A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics
• The reliability of replications: a study in computational reproductions
• Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
• Examining the replicability of online experiments selected by a decision market
• Promoting Reproducibility and Replicability in Political Science
• Toward a more credible assessment of the credibility of science by many-analyst studies
• The Uptake of Preregistration, and Its Impact on the Proportion of Supported Hypotheses, in Sports Science
• Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy
• Investing in open and FAIR practices for more usable and equitable climate-risk research
• Care to share? Experimental evidence on code sharing behavior in the social sciences. [europepmc]
• On the scope of scientific hypotheses. [europepmc]
• The positive-negative-competence (PNC) model of psychological responses to representations of robots. [europepmc]
• Grilling the data: application of specification curve analysis to red meat and all-cause mortality. [europepmc]
• Challenges in the Use of AI-Driven Non-Destructive Spectroscopic Tools for Rapid Food Analysis. [europepmc]
• Unraveling the genetics of arsenic toxicity with cellular morphology QTL. [europepmc]
• Subjective evidence evaluation survey for many-analysts studies. [europepmc]
• The replication crisis has led to positive structural, procedural, and community changes. [europepmc]
• We urgently need a culture of multi-operationalization in psychological research. [europepmc]
• The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche. [europepmc]
• Overcoming the pitfalls of categorizing continuous variables in ecology, evolution and behaviour. [europepmc]
• Reproducibility and interpretability in radiomics: a critical assessment [europepmc]
• Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. [europepmc]
• The reliability of replications: a study in computational reproductions. [europepmc]
• The hidden complexity of the simple world of basic experimental psychology: the principal and practical limits of gaining psychological knowledge using the experimental method. [europepmc]
• Marital status and risk of cardiovascular disease - a multi-analyst study in epidemiology. [europepmc]
• Big team science reveals promises and limitations of machine learning efforts to model physiological markers of affective experience. [europepmc]
• A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics. [europepmc]
• Robust statistical methods and the credibility movement of psychological science. [europepmc]
• The Systematic Multiverse Analysis Registration Tool for defining multiverse analyses. [europepmc]
• SciSciGPT: advancing human-AI collaboration in the science of science. [europepmc]
• Sensitivity Analysis: A Practical Guide to Teaching. [europepmc]
• Toward a paradox-oriented paradigm of knowing: the form theory of observation and quality in empirical sciences. [europepmc]
• Step-by-step causal analysis of Electronic Health Records to ground decision making [europepmc]
• One data set, many analysts: Implications for practicing scientists. [europepmc]

Discussions

• That's false. It is not what the paper shows. First of all, the "bias" is minuscule. Quoting Borjas's co-author's analysis of the same dataset: "Researchers' expertise, prior beliefs, and expect [bsky, 79 points, 2 comments]
• doi.org/10.1073/pnas... is an example from another field of results varying with teams analysing the data as reported in fMRI (www.nature.com/articles/s41...). Is multiverse analysis the way to go or [bsky, 0 points, 0 comments]

Related

• Why Most Published Research Findings Are False
• Toward a more credible assessment of the credibility of science by many-analyst studies
• The Paradox of Collective Certainty in Science
• The natural selection of bad science
• The Perils of Advocacy