Meaningful Associations Redux: Quantifying and interpreting effect size in the context of the Adolescent Brain and Cognitive Development study

2025/10/11 by Anthony Steven Dick, Jonathan S. Comer, Mohammadreza Bayat +7 · 1 voice · 1 citation
Psychology · Decision Sciences · #Cognitive Abilities and Testing #Psychometric Methodologies and Testing #Meta-analysis and systematic reviews

paper · doi:10.1016/j.dcn.2025.101630

Abstract

The Adolescent Brain Cognitive Development (ABCD) Study represents a pioneering initiative that aims to unravel the complexities of behavioral and neural development in youth. In this paper, we address the challenges inherent in extracting meaningful insights from the extensive data compiled by the ABCD initiative. Our focus is on advocating for best practices in reproducible research, interpretation of effect size, and reporting of principled results. Central to this discourse is a detailed examination of effect sizes within the expansive ABCD dataset, and how they can be meaningfully interpreted in the context of large-scale research. We describe the hurdles associated with transitioning from conventional small-sample studies to the opportunities and challenges of large samples, including the phenomenon of statistically significant but practically trivial effects. To promote transparent and rigorous inference, we present a four-part framework to evaluate observed effects: researchers should define a smallest effect size of interest (SESOI), compare estimates to relevant benchmarks, test whether observed effects exceed meaningful thresholds (e.g., through equivalence testing), and visualize results to enhance interpretation and communication. Applying this framework yields a clearer, more cumulative understanding of effect size interpretation and contributes substantively to the refinement of scientific practices within adolescent brain and cognitive development research.

Citations

• Estimating the reproducibility of psychological science
• Small Sample Inference for Fixed Effects from Restricted Maximum Likelihood
• The file drawer problem and tolerance for null results.
• Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance
• Interpreting the early language trajectories of children from low-SES and language minority homes: Implications for closing achievement gaps.
• The ASA Statement on p -Values: Context, Process, and Purpose
• Equivalence Testing for Psychological Research: A Tutorial
• Empirically derived guidelines for effect size interpretation in social psychology
• When (not) to rely on the reliable change index: A critical appraisal and alternatives to consider in clinical psychology.
• Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
• Vaccines are not associated with autism: An evidence-based meta-analysis of case-control and cohort studies
• P-curve: A key to the file-drawer.
• Why Most Discovered True Associations Are Inflated
• Parkinson's Disease
• Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It

Cited by

• Individual differences in adolescent cortical development are associated with neighborhood characteristics: Longitudinal findings from the ABCD study

Discussions

• Here's a nice discussion of effect sizes in cognitive neuroscience, with real examples and an discussion of using null intervals. Lots of good advice here; if you've been interested in inference by i [bsky, 7 points, 1 comments]

Related

• Reproducible brain-wide association studies require thousands of individuals
• Longitudinal analysis of the ABCD® study
• Evaluating Effect Size in Psychological Research: Sense and Nonsense
• Beyond Increasing Sample Sizes: Optimizing Effect Sizes in Neuroimaging Research on Individual Differences
• Correlational effect size benchmarks.