Model-Based Measures for Detecting and Quantifying Response Bias

2018/06/15 by R. Philip Chalmers · 28 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Causal Inference Techniques #Advanced Statistical Modeling Techniques #Categorical variable #Computer science #Differential item functioning #Econometrics #Item response theory #Marginal likelihood #Mathematics #Maximum likelihood #Monte Carlo method #Psychometric Methodologies and Testing #Psychometrics #Response bias #Sample size determination #Statistical hypothesis testing #Statistics

paper · doi:10.1007/s11336-018-9626-9

published in Psychometrika 83(3), 696-732 (Springer Science+Business Media)

openalex publication_date 2018/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

This paper proposes a model-based family of detection and quantification statistics to evaluate response bias in item bundles of any size. Compensatory (CDRF) and non-compensatory (NCDRF) response bias measures are proposed, along with their sample realizations and large-sample variability when models are fitted using multiple-group estimation. Based on the underlying connection to item response theory estimation methodology, it is argued that these new statistics provide a powerful and flexible approach to studying response bias for categorical response data over and above methods that have previously appeared in the literature. To evaluate their practical utility, CDRF and NCDRF are compared to the closely related SIBTEST family of statistics and likelihood-based detection methods through a series of Monte Carlo simulations. Results indicate that the new statistics are more optimal effect size estimates of marginal response bias than the SIBTEST family, are competitive with a selection of likelihood-based methods when studying item-level bias, and are the most optimal when studying differential bundle and test bias.

Citations

• It Might Not Make a Big DIF
• Marginal Maximum Likelihood Estimation of Item Parameters: Application of an EM Algorithm
• A Model-Based Standardization Approach that Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DTF as well as Item Bias/DIF
• The Area between Two Item Characteristic Curves

Cited by

• Out-of-Level Cognitive Testing of Children with Special Educational Needs
• Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects
• Examining the measurement equivalence of the Maslach Burnout Inventory across age, gender, and specialty groups in US physicians. [europepmc]
• Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS®) Measures: Methods, Challenges, Advances, and Future Directions. [europepmc]
• An R toolbox for score-based measurement invariance tests in IRT models. [europepmc]
• Item response theory and differential test functioning analysis of the HBSC-Symptom-Checklist across 46 countries. [europepmc]
• Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects. [europepmc]
• Including Empirical Prior Information in the Reliable Change Index. [europepmc]
• The Developmental Assessment of Social Communication Ability (DASCA): initial creation and psychometric description. [europepmc]
• Assessing the psychometric properties of the Youth Externalizing Problems Screener (YEPS) in peruvian adolescents. [europepmc]

Related

• Item Response Thresholds Models
• A semiparametric approach for item response function estimation to detect item misfit
• A Model-Based Standardization Approach that Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DTF as well as Item Bias/DIF
• Comparing item response theory models for ranking incorrect response options
• A novel nonparametric framework for DIF detection using kernel-smoothed item response curves