LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment

2026/06/30 by Han Chen, Ming Li, Chenguang Wang +4 · 2 citations
Computer Science · #cs.CL

paper · pdf

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Existing work on LLM-based educational assessment has focused largely on item difficulty, but difficulty alone does not indicate whether an item meaningfully distinguishes higher- from lower-proficiency students. Item discrimination captures this complementary and fundamental psychometric property. We investigate whether LLMs can predict human item discrimination from assessment content. We evaluate 42 proprietary and open-weight LLMs using two complementary approaches. Direct discrimination prediction asks models to explicitly predict an item's discrimination value, while response-based proxy estimation treats LLM answers as synthetic responses and applies a Classical Test Theory (CTT)-inspired item-rest calculation. Direct predictions show weak alignment with human item discrimination. The response-based proxy provides a stronger but still limited ranking signal, reaching a CEFR-stratified rank correlation of 0.231. Further analysis shows that this correlation comes mainly from differences across models rather than proficiency prompts that reliably simulate students at different ability levels. Current LLMs therefore contain some discrimination-relevant information, but they do not yet reliably model the ability-conditioned human response behavior that gives item discrimination its psychometric meaning.

Citations

• Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors
• Modeling Student Learning with 3.8 Million Program Traces
• Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review
• ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle
• Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?
• Qwen3 Technical Report
• Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?
• QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
• Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs
• Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation
• LLM-based Cognitive Models of Students with Misconceptions
• The Llama 3 Herd of Models
• Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution
• Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
• A quantitative study of NLP approaches to question difficulty estimation
• Generative Agents: Interactive Simulacra of Human Behavior
• Teaching Models to Express Their Uncertainty in Words
• AutoTutor: An Intelligent Tutoring System With Mixed-Initiative Dialogue
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling