Controlling Cloze-test Question Item Difficulty with PLM-based Surrogate Models for IRT Assessment

2024/03/03 by Jingshen Zhang, Zhang, Jingshen, Jiajun Xie +3 · 1 citation
Computer Science · Mathematics · Psychology · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer science #Computers and Society (cs.CY) #Econometrics #Educational Technology and Assessment #FOS: Computer and information sciences #Mathematics #Natural language processing #Psychology #Test (biology)

paper · pdf · doi:10.48550/arxiv.2403.01456

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/03/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Item difficulty plays a crucial role in adaptive testing. However, few works have focused on generating questions of varying difficulty levels, especially for multiple-choice (MC) cloze tests. We propose training pre-trained language models (PLMs) as surrogate models to enable item response theory (IRT) assessment, avoiding the need for human test subjects. We also propose two strategies to control the difficulty levels of both the gaps and the distractors using ranking rules to reduce invalid distractors. Experimentation on a benchmark dataset demonstrates that our proposed framework and methods can effectively control and evaluate the difficulty levels of MC cloze tests.

Related

• Difficulty-Controllable Cloze Question Distractor Generation
• SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction
• Introducing a framework to assess newly created questions with Natural Language Processing
• Reliable and Efficient Amortized Model-based Evaluation
• Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models