2026/07/21 by Emily Zhou, Joseph Holler, Peter Kedron
Decision Sciences · Computer Science · #Scientific Computing and Data Management #Software Engineering Research #Software Engineering Techniques and Practices
paper · doi:10.1111/gean.70053
ABSTRACT Researchers increasingly call for transparent reporting of analytic decisions as foundations for reproducible research and collective knowledge production. However, these calls for openness often overlook a critical dimension: software environments encode implicit computational decisions that are not always visible to researchers, even when the same statistical methods are applied. Analyses may be technically reproducible, while the epistemological function of reproduction to evaluate and validate prior conclusions is undermined by the opacity of software design. Moreover, unexamined software behavior can compromise the conceptual rigor of research by shaping analytic choices unbeknownst to researchers. While attempting to reproduce a study correlating spatial distributions of COVID‐19 and people with disabilities, we investigated how software‐encoded decisions influenced intermediate analytic outputs, propagated into downstream inference, and ultimately shaped the research design. Through systematic comparison across software, we demonstrate that (1) software‐encoded decisions structure both intermediate results and final analytic conclusions; (2) researchers must remain attentive to the decisions and materials required to execute an analysis, understand what software is doing on their behalf, and consider the numerical and conceptual ramifications of those decisions; and (3) cross‐software sensitivity analysis is particularly important when analytic workflows span multiple platforms or when outputs from one system are used as inputs to another.