Using anchor-based methods to determine the smallest effect size of interest

2021/05/18 by Farid Anvari, Daniël Lakens · 2 voices · 41 citations
Psychology · #Behavioral Health and Interventions #Mental Health Research Topics #Psychological Well-being and Life Satisfaction

paper · doi:10.1016/j.jesp.2021.104159

openalex publication_date 2021/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Effect sizes are an important outcome of quantitative research, but few guidelines exist that explain how researchers can determine which effect sizes are meaningful. Psychologists often want to study effects that are large enough to make a difference to people's subjective experience. Thus, subjective experience is one way to gauge the meaningfulness of an effect. We propose and illustrate one method for how to quantify the smallest subjectively experienced difference—the smallest change in an outcome measure that individuals consider to be meaningful enough in their subjective experience such that they are willing to rate themselves as feeling different—using an anchor-based method with a global rating of change question applied to the positive and negative affect scale. We provide a step-by-step guide for the questions that researchers need to consider in deciding whether and how to use the anchor-based method, and we make explicit the assumptions of the method that future research can examine. For researchers interested in people's subjective experiences, this anchor-based method provides one way to specify a smallest effect size of interest, which allows researchers to interpret observed results in terms of their theoretical and practical significance.

Citations

• Equivalence Testing for Psychological Research: A Tutorial
• Development and validation of brief measures of positive and negative affect: The PANAS scales.
• Interpretation of Changes in Health-related Quality of Life
• The Problem of Units and the Circumstance for POMP
• Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes
• Statistical power analysis for the behavioral sciences
• Standardized or simple effect size: What should be reported?
• The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network
• Invisible Hands and Fine Calipers: A Call to Use Formal Theory as a Toolkit for Theory Construction

Cited by

• A Tutorial on Analyzing Ecological Momentary Assessment Data in Psychological Research With Bayesian (Generalized) Mixed-Effects Models
• From significance testing to estimation and Open Science: How esci can help
• Facebook increases political knowledge, reduces well-being and informational treatments do little to help
• The Effect of Different Construct Operationalizations, Study Duration, and Preprocessing Choices on Power-Based Sample Size Recommendations in Intensive Longitudinal Research
• The Extended Morality as Cooperation Dictionary (eMACD): A Crowd-Sourced Approach via the Moral Narrative Analyzer Platform
• The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment
• What's wrong with rating scales? Psychology's replication and confidence crisis cannot be solved without transparency in data generation
• How Many Participants Do I Need to Test an Interaction? Conducting an Appropriate Power Analysis and Achieving Sufficient Power to Detect an Interaction
• Three objections to a novel paradigm in social media effects research
• From arbitrary metrics to meaningful change: Empirical smallest effect sizes of interest for the Feeling Scale during exercise
• Globally, songs and instrumental melodies are slower and higher and use more stable pitches than speech: A Registered Report
• Teenagers, screens and social media: a narrative review of reviews and key studies
• Metrological uncertainty and the value of informative nulls in marketing research
• Effectiveness of a Mobile-Based Self-Regulation Training on Youths’ Affect
• Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability
• Regional variation in nitric oxide‐dependent cutaneous vasodilatation during local heating in young adults
• Practical routes to preregistration: a guide to enhanced transparency and rigour in neuropsychological research
• Time spent playing video games is unlikely to impact well-being. [europepmc]
• Are Small Effects the Indispensable Foundation for a Cumulative Psychological Science? A Reply to Götz et al. (2022). [europepmc]
• Smartphone video nystagmography using convolutional neural networks: ConVNG. [europepmc]
• Power analysis for conditional indirect effects: A tutorial for conducting Monte Carlo simulations with categorical exogenous variables. [europepmc]
• Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field. [europepmc]
• The importance of the smallest effect size of interest in expert witness testimony on alcohol and memory. [europepmc]
• A Digital Single-Session Intervention Platform for Youth Mental Health: Cultural Adaptation, Evaluation, and Dissemination. [europepmc]
• Implementing a velocity-based approach to resistance training: the reproducibility and sensitivity of different velocity monitoring technologies. [europepmc]
• Effect of environmental enrichment on relapse rates in patients with severe alcohol use disorder: protocol for a randomised controlled trial. [europepmc]
• The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment. [europepmc]
• Neural responses to natural and enhanced speech edges in children with and without dyslexia. [europepmc]
• Experimental Therapeutics: Opportunities and Challenges Stemming From the National Institute of Mental Health Workshop on Novel Target Discovery and Psychosocial Intervention Development. [europepmc]
• Recognizing limits on the generalizability of findings of psychological science research. [europepmc]
• Power to Detect What? Considerations for Planning and Evaluating Sample Size. [europepmc]
• Validation and application of computer vision algorithms for video-based tremor analysis. [europepmc]
• Estimating substance use disparities across intersectional social positions using machine learning: An application of group-lasso interaction network. [europepmc]
• Applying a synergistic mindsets intervention to an esports context. [europepmc]
• Estimating power in complex nonlinear structural equation modeling including moderation effects: The powerNLSEM R-package. [europepmc]
• Facebook increases political knowledge, reduces well-being and informational treatments do little to help. [europepmc]
• Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting. [europepmc]
• Identifying the Minimal Clinically Important Difference in Emotion Regulation Among Youth Using the JoyPop App: Survey Study. [europepmc]
• A Tutorial on Analyzing Ecological Momentary Assessment Data in Psychological Research With Bayesian (Generalized) Mixed-Effects Models. [europepmc]
• Ready to ROC? A tutorial on simulation-based power analyses for null hypothesis significance, minimum-effect, and equivalence testing for ROC curve analyses. [europepmc]
• Rapid cultural adaptations for scalable dissemination of a single-session intervention among Polish and Ukrainian youth: An open pilot trial. [europepmc]
• Web-Based Single Session Intervention for Perceived Control Over Anxiety During COVID-19: Randomized Controlled Trial. [europepmc]

Discussions

• I love this paper from @faridanvari.bsky.social trying to figure out the smallest effect size (for self-reports such as affect) that people can subjectively experience doi.org/10.1016/j.je... [bsky, 7 points, 0 comments]
• The most common approach is an anchor-based approach (i.e., using an external anchor/reference variable as a guidance to derive a SESOI), especially for psychological variables that have no natural me [bsky, 1 points, 1 comments]

Related

• Evaluating Effect Size in Psychological Research: Sense and Nonsense
• Interpreting effect sizes: Toward a quantitative cumulative social psychology
• The use and interpretation of unstandardized and standardized effect sizes in psychology: current practices and challenges
• Empirically derived guidelines for effect size interpretation in social psychology
• Standardized or simple effect size: What should be reported?