Insufficient effort responding: Examining an insidious confound in survey data.
2014/12/15 by Jason L. Huang, Mengqiao Liu, Nathan A. Bowling · 40 citations
Business, Management and Accounting · Social Sciences · #Job Satisfaction and Organizational Behavior #Customer Service Quality and Loyalty #Experimental Behavioral Economics Studies
paper · doi:10.1037/a0038510
Abstract
Insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) to surveys has largely been assumed to be a source of random measurement error that attenuates associations between substantive measures. The current article, however, illustrates how and when the presence of IER can produce a systematic bias that inflates observed correlations between substantive measures. Noting that inattentive responses as a whole generally congregate around the midpoint of a Likert scale, we propose that Mattentive, defined as the mean score of attentive respondents on a substantive measure, will be negatively related to IER's confounding effect on substantive measures (i.e., correlations between IER and a given substantive measure will become less positive [or more negative] as Mattentive increases). Results from a personality questionnaire (Study 1) and a simulation (Study 2) consistently support the hypothesized confounding influence of IER. Using an employee sample (Study 3), we demonstrated how IER can confound bivariate relationships between substantive measures. Together, these studies indicate that IER can inflate the strength of observed relationships when scale means depart from the scale midpoints, resulting in an inflated Type I error rate. This challenges the traditional view that IER attenuates observed bivariate correlations. These findings highlight situations where IER may be a methodological nuisance, while underscoring the need for survey administrators and researchers to deter and detect IER in surveys. The current article serves as a wake-up call for researchers and practitioners to more closely examine IER in their data.
Cited by
- Methods for the detection of carelessly invalid responses in survey data
- Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices
- Partisan Expressive Responding: Lessons from Two Decades of Research
- Careless Responding: Why Many Findings Are Spurious or Spuriously Inflated
- Robust Estimation of Polychoric Correlation
- Integrating Personality Structure, Personality Process, and Personality Development
- Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations
- Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA
- Validating common-sense survey items as a priori indicators of careless responses: Evidence from construct, convergent, and concurrent validity
- An Investigation of the Temporal Dynamics of Careless Responding Across Different Populations in Experience Sampling Data
- Navigating autonomy and control in human-AI delegation: User responses to technology- versus user-invoked task allocation
- Robust Estimation of Polyserial Correlation
- Attention checks and how to use them: Review and practical recommendations
- Evaluating Methods to Prevent and Detect Inattentive Respondents in Web Surveys
- Dependent latent class partial credit models for careless and insufficient effort responding in online survey data
- Satisficing in Mental Health Care Patients: The Effect of Cognitive Symptoms on Self-Report Data Quality. [europepmc]
- Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations. [europepmc]
- Random responses inflate statistical estimates in heavily skewed addictions data. [europepmc]
- Balanced and positively worded personality short-forms: Mini-IPIP validity and cross-cultural invariance. [europepmc]
- Careless Responding Threatens Factorial Analytic Results and Construct Validity of Personality Measure. [europepmc]
- Methods of Detecting Insufficient Effort Responding: Comparisons and Practical Recommendations. [europepmc]
- Self-employment and eudaimonic well-being: Energized by meaning, enabled by societal legitimacy. [europepmc]
- How does COVID-19 emergency cognition influence public pro-environmental behavioral intentions? An affective event perspective. [europepmc]
- Dismissing "Don't Know" Responses to Perceived Risk Survey Items Threatens the Validity of Theoretical and Empirical Behavior-Change Research. [europepmc]
- A Response-Time-Based Latent Response Mixture Model for Identifying and Modeling Careless and Insufficient Effort Responding in Survey Data. [europepmc]
- Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting. [europepmc]
- Quality of Survey Responses at Older Ages Predicts Cognitive Decline and Mortality Risk. [europepmc]
- Reliability Generalization Meta-Analysis of Internet Gaming Disorder Scale. [europepmc]
- Real-world data using mHealth apps in rhinitis, rhinosinusitis and their multimorbidities. [europepmc]
- Data Quality and Study Compliance Among College Students Across 2 Recruitment Sources: Two Study Investigation. [europepmc]
- Measuring the suicidal mind: The 'open source' Suicidality Scale, for adolescents and adults. [europepmc]
- Accounting for careless and insufficient effort responding in large-scale survey data-development, evaluation, and application of a screen-time-based weighting procedure. [europepmc]
- The effect of credence attributes on willingness to pay a premium for organic food: A moderated mediation model of attitudes and uncertainty. [europepmc]
- Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. [europepmc]
- Testing Explanations for Skepticism of Personalized Risk Information. [europepmc]
- Syndromes associated with frontotemporal lobar degeneration change response patterns on visual analogue scales. [europepmc]
- Inattentive responding can induce spurious associations between task behaviour and symptom measures. [europepmc]
- Detecting non-content-based response styles in survey data: An application of mixture factor analysis. [europepmc]
- Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19 Study. [europepmc]
- Burden and Inattentive Responding in a 12-Month Intensive Longitudinal Study: Interview Study Among Young Adults. [europepmc]
Related