Random Responding as a Threat to the Validity of Effect Size Estimates in Correlational Research

2010/04/28 by Marcus Credé · 169 citations
Mathematics · Psychology · Social Sciences · #Affect (linguistics) #Econometrics #Mathematics #Mental Health Research Topics #Meta-analysis #Nomothetic #Nomothetic and idiographic #Personality Traits and Psychology #Psychology #Random effects model #Social and Intergroup Psychology #Social psychology #Statistics

paper · doi:10.1177/0013164410366686

published in Educational and Psychological Measurement 70(4), 596-612 (SAGE Publishing)

openalex publication_date 2010/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/26

Abstract

Random responding to psychological inventories is a long-standing concern among clinical practitioners and researchers interested in interpreting idiographic data, but it is typically viewed as having only a minor impact on the statistical inferences drawn from nomothetic data. This article explores the impact of random responding on the size and direction of correlations observed between multi-item inventory scores. Random responses to individual items result in nonrandomly distributed inventory-level scores. Therefore, even low base rates of random responding can significantly affect the statistical inferences made from inventory-level data. Study 1 uses simulations to show that even low base rates of random responding can significantly affect observed correlations, especially when the inventories in question assess low or high base rate phenomena. Study 2 uses archival data to illustrate the moderating effect of random responding on observed correlations in two samples.

Cited by

• The Impact of Attentiveness Interventions on Survey Data
• Careless Responding: Why Many Findings Are Spurious or Spuriously Inflated
• Careless responding detection revisited: Accuracy of direct and indirect measures
• Robust Estimation of Polychoric Correlation
• Integrating Personality Structure, Personality Process, and Personality Development
• Attention checks and how to use them: Review and practical recommendations
• Using multi-rater and test-retest data to detect overlap within and between psychological scales
• Dependent latent class partial credit models for careless and insufficient effort responding in online survey data
• A Tutorial on What to Do With Skewness, Kurtosis, and Outliers: New Insights to Help Scholars Conduct and Defend Their Research
• Refining Measures for Assessing Problematic/Addictive Digital Gaming Use in Clinical and Research Settings. [europepmc]
• Using Person Fit Statistics to Detect Outliers in Survey Research. [europepmc]
• Satisficing in Mental Health Care Patients: The Effect of Cognitive Symptoms on Self-Report Data Quality. [europepmc]
• Random responses inflate statistical estimates in heavily skewed addictions data. [europepmc]
• Careless Responding Threatens Factorial Analytic Results and Construct Validity of Personality Measure. [europepmc]
• Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting. [europepmc]
• Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey. [europepmc]
• The utility of the integrated behavior change model as an extension of the theory of planned behavior. [europepmc]
• Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation. [europepmc]
• Supervised Classes, Unsupervised Mixing Proportions: Detection of Bots in a Likert-Type Questionnaire. [europepmc]
• The PCQ-Infertility Revised: A New Digital Instrument to Measure Treatment Satisfaction of Fertility Patients. [europepmc]
• Using heterogeneous sources of data and interpretability of prediction models to explain the characteristics of careless respondents in survey data. [europepmc]
• Burden and Inattentive Responding in a 12-Month Intensive Longitudinal Study: Interview Study Among Young Adults. [europepmc]
• Careless responding detection revisited: Accuracy of direct and indirect measures. [europepmc]
• A comparison of conventional and resampled personal reliability in detecting careless responding. [europepmc]
• Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires. [europepmc]
• The Impact of Insufficient Effort Responses on the Order of Category Thresholds in the Polytomous Rasch Model. [europepmc]
• AI-powered fraud and the erosion of online survey integrity: an analysis of 31 fraud detection strategies. [europepmc]
• The Impact of Attentiveness Interventions on Survey Data. [europepmc]
• Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024. [europepmc]
• Reducing Calibration Bias for Person Fit Assessment by Mixture Model Expansion. [europepmc]
• Robust Estimation of Polychoric Correlation. [europepmc]
• More than just noise: careless responding and its systematic effects on reliability, validity, and measurement invariance. [europepmc]

Related

• Correlational effect size benchmarks.
• Multiple Trials May Yield Exaggerated Effect Size Estimates
• Understanding Correlation: Factors That Affect the Size of r
• Significance and Replication in simple counting experiments: Distributional Null Hypothesis Testing
• Small is beautiful: In defense of the small-N design