A Multilevel Mixture IRT Framework for Modeling Response Times as Predictors or Indicators of Response Engagement in IRT Models

2021/09/13 by Gabriel Nagy, Esther Ulitzsch · 49 citations
Mathematics · Psychology · #Behavioral Health and Interventions #Cognitive Abilities and Testing #Computer science #Developmental psychology #Econometrics #Flexibility (engineering) #Item response theory #Latent class model #Latent variable #Latent variable model #Machine learning #Mathematics #Mental Health Research Topics #Multilevel model #Psychology #Psychometrics #Response time #Scale (ratio) #Statistics

paper · pdf · doi:10.1177/00131644211045351

published in Educational and Psychological Measurement 82(5), 845-879 (SAGE Publishing)

openalex publication_date 2021/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

Disengaged item responses pose a threat to the validity of the results provided by large-scale assessments. Several procedures for identifying disengaged responses on the basis of observed response times have been suggested, and item response theory (IRT) models for response engagement have been proposed. We outline that response time-based procedures for classifying response engagement and IRT models for response engagement are based on common ideas, and we propose the distinction between independent and dependent latent class IRT models. In all IRT models considered, response engagement is represented by an item-level latent class variable, but the models assume that response times either reflect or predict engagement. We summarize existing IRT models that belong to each group and extend them to increase their flexibility. Furthermore, we propose a flexible multilevel mixture IRT framework in which all IRT models can be estimated by means of marginal maximum likelihood. The framework is based on the widespread Mplus software, thereby making the procedure accessible to a broad audience. The procedures are illustrated on the basis of publicly available large-scale data. Our results show that the different IRT models for response engagement provided slightly different adjustments of item parameters of individuals' proficiency estimates relative to a conventional IRT model.

Cited by

• Dependent latent class partial credit models for careless and insufficient effort responding in online survey data
• A Response-Time-Based Latent Response Mixture Model for Identifying and Modeling Careless and Insufficient Effort Responding in Survey Data. [europepmc]
• The Role of Response Times on the Measurement of Mental Ability. [europepmc]
• Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating. [europepmc]
• Accounting for careless and insufficient effort responding in large-scale survey data-development, evaluation, and application of a screen-time-based weighting procedure. [europepmc]
• Testing Replicability and Generalizability of the Time on Task Effect. [europepmc]
• State-Aware Deep Item Response Theory using student facial features. [europepmc]
• Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators. [europepmc]
• A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior. [europepmc]
• The Impact of Insufficient Effort Responses on the Order of Category Thresholds in the Polytomous Rasch Model. [europepmc]
• Is Effort Moderated Scoring Robust to Multidimensional Rapid Guessing? [europepmc]
• Accounting for Persistence in Tests with Linear Ballistic Accumulator Models. [europepmc]
• Comparing Different Approaches of (Not) Accounting for Rapid Guessing in Plausible Values Estimation. [europepmc]
• Misclassification Produced by Rapid-Guessing Identification Methods and Their Suitability Under Various Conditions. [europepmc]
• A Two-Step Method Based on lz* for Identifying Effortful Respondents. [europepmc]
• Faking in High-Stakes Personality Assessments: A Response-Time-Based Latent Response Mixture Modeling Approach. [europepmc]

Related

• An Application of Item Response Time: The Effort‐Moderated IRT Model
• Item Response Theory Models for Performance Decline During Testing
• Mixture IRT Model With a Higher-Order Structure for Latent Traits
• Response Mixture Modeling: Accounting for Heterogeneity in Item Characteristics across Response Times
• Response moderation models for conditional dependence between response time and response accuracy