Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
2020/12/01 by Jessica Kay Flake, Eiko I. Fried · 2 voices · 44 citations
Decision Sciences · Psychology · #Mental Health Research Topics #Meta-analysis and systematic reviews #Psychometric Methodologies and Testing
paper · doi:10.1177/2515245920952393
openalex publication_date 2020/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
Abstract
In this article, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons, including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical-conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, and pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.
Citations
Cited by
- Linguistic alignment with an artificial agent: A commentary and re-analysis
- Beyond vernacular: Measurement solutions to the lexical fallacy in disgust research
- The case for formal methodology in scientific reform
- Beyond binary group categorization: towards a dynamic view of human groups
- Psychometric Properties of Eating Behaviour Instruments for Older Adults: A Systematic Review
- Commentary: Prediction of problem gambling by demographics, gaming behavior and psychological correlates among gacha gamers: A cross-sectional online survey in Chinese young adults
- Registered report: Survey on attitudes and experiences regarding preregistration in psychological research
- What’s the difference between measures of wellbeing, quality of life, resilience, and coping? An umbrella review and concept map of 155 measures of positive mental health
- Conflation between self-report and neurocognitive assessments of cognitive flexibility: a critical review of the Jingle Fallacy
- What we mean when we say semantic: Toward a multidisciplinary semantic glossary
- New developments in experience sampling methodology
- The replication crisis has led to positive structural, procedural, and community changes
- How measurement affects causal inference: Attenuation bias is (usually) more important than outcome scoring weights
- Risk Psychology & Cyber-Attack Tactics
- Theories and Models: What They Are, What They Are for, and What They Are About
- What's wrong with rating scales? Psychology's replication and confidence crisis cannot be solved without transparency in data generation
- ‘We're not going to start lifting stones now…’: Stakeholder perspectives on the role of psychometric methods in outcome measurement
- Black Summer arson: Examining the impact of climate misinformation and corrections on reasoning
- Sample-Size Planning in Item-Response Theory: A Tutorial
- Construct validity evidence reporting practices for the Reading the Mind in the Eyes Test: A systematic scoping review
- Towards better hypothesis tests in oxytocin research: Evaluating the validity of auxiliary assumptions
- Valid and Reliable? Basic and Expanded Recommendations for Psychometric Reporting and Quality Assessment.
- A Plea to Psychology Professional Societies that Publish Journals: Assess Computational Reproducibility
- Introduction and Validation of the Short Antinatalism Scale (S-ANS)
- Psychometric validation and refinement of the Interoception Sensory Questionnaire (ISQ) in adolescents and adults on the autism spectrum
- A systematic review of interpersonal processes and their measurement within experience sampling studies of self-injurious thoughts and behaviours
- Enhancing Intellectual Humility About Political Topics: An Intervention Tournament Including Five Conceptual Replications
- Reliability representativeness: How well does coefficient alpha summarize reliability across the score distribution?
- Reading the Mind in the Eyes Test Scores Demonstrate Poor Structural Properties in Nine Large Non-Clinical Samples
- Psychology Is a Property of Persons, Not Averages or Distributions: Confronting the Group-to-Person Generalizability Problem in Experimental Psychology
- Using multi-rater and test-retest data to detect overlap within and between psychological scales
- Examining item content across nine psychological (in)flexibility scales: What do they measure?
- The measurement of goal dimensions: A critical review.
- Improving Social Science: Lessons from the Open Science Movement
- Long-term word frequency dynamics derived from Twitter are corrupted: A bespoke approach to detecting and removing pathologies in ensembles of time series
- Short and long-term dominance of negative information in shaping public energy perceptions: The case of shallow geothermal systems
- The dire disregard of measurement invariance testing in psychological science.
- What Can Educational Psychology Learn From, and Contribute to, Theory Development Scholarship?
- Measures of Relationship Power Dynamics in Romantic Relationships
- Multidimensional Signals and Analytic Flexibility: Estimating Degrees of Freedom in Human-Speech Analyses
- Psychology's Theory Crisis, and Why Formal Modelling Cannot Solve It
- Does Truth Pay? Investigating the Effectiveness of the Bayesian Truth Serum With an Interim Payment: A Registered Report
- Conservatism in response to rising universalism? Linking individuals' conservation values, right‐wing attitudes, and nationalism to perceived prevalence of universalism among fellow citizens
- A Guide for Calculating Study-Level Statistical Power for Meta-Analyses
- Normalizing the Use of Single-Item Measures: Validation of the Single-Item Compendium for Organizational Psychology
- Estimating Heterogeneous Treatment Effects with Item-Level Outcome Data: Insights from Item Response Theory
Discussions
Related