2026/01/24 by Annika Iris Külpmann, Jan-Paul Ries, Ian Hussey +1 · 1 voice
Decision Sciences · Neuroscience · Psychology · #Meta-analysis and systematic reviews #Neural and Behavioral Psychology Studies #Sport Psychology and Performance
paper · doi:10.31234/osf.io/4g3vr_v1
openalex publication_date 2026/01/24 · openalex created_date 2026/01/25 · openalex updated_date 2026/07/14
Methodological flexibility–researcher degrees of freedom in procedure and scoring–can contribute to poor measurement and low replicability of findings. To quantify methodological flexibility in the Iowa Gambling Task (IGT), a widely used measure of decision-making in clinical psychology, we conducted a meta-method review of a random sample of 100 studies. We found prevalent under-reporting and wide-spread modifications of procedural details. We identified 244 distinct scores used to quantify IGT outcomes, 177 of which appeared only once in our sample. To quantify the potential impact of this flexibility, we performed a multiverse analysis using 205 scores on an open IGT dataset. Correlations among these ranged from rs = -0.942 to 0.998 with a median r = 0.022, suggesting that different scoring strategies can produce extremely different study outcomes. Given these findings, without substantive information about which specific procedural and scoring choices were employed, results from the IGT are rendered uninterpretable. Supplementary Materials, Data, and Code are available at github.com/anniria/igt-methodological-flexibility.