Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?

2025/06/11 by Andreas Säuberli, Säuberli, Andreas, Diego Frassinelli +3 · 3 citations
Computer Science · Decision Sciences · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning #Psychometric Methodologies and Testing #Student Assessment and Feedback

paper · pdf · doi:10.48550/arxiv.2506.09796

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Knowing how test takers answer items in educational assessments is essential for test development, to evaluate item quality, and to improve test validity. However, this process usually requires extensive pilot studies with human participants. If large language models (LLMs) exhibit human-like response behavior to test items, this could open up the possibility of using them as pilot participants to accelerate test development. In this paper, we evaluate the human-likeness or psychometric plausibility of responses from 18 instruction-tuned LLMs with two publicly available datasets of multiple-choice test items across three subjects: reading, U.S. history, and economics. Our methodology builds on two theoretical frameworks from psychometrics which are commonly used in educational assessment, classical test theory and item response theory. The results show that while larger models are excessively confident, their response distributions can be more human-like when calibrated with temperature scaling. In addition, we find that LLMs tend to correlate better with humans in reading comprehension items compared to other subjects. However, the correlations are not very strong overall, indicating that LLMs should not be used for piloting educational assessments in a zero-shot setting.

Citations

• Instruction‐Tuned Large‐Language Models for Quality Control in Automatic Item Generation: A Feasibility Study
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Large Language Models Are Not Robust Multiple Choice Selectors
• Analyzing Multiple-Choice Reading and Listening Comprehension Tests
• Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution
• Stop Measuring Calibration When Humans Disagree
• Multiple-Choice Question Generation: Towards an Automated Assessment Framework
• Predicting Human Psychometric Properties Using Computational Language Models
• Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
• On Calibration of Modern Neural Networks

Cited by

• Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
• Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach
• Opportunities and Challenges of LLMs in Education: An NLP Perspective

Related

• Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?
• Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models
• Psychometric Predictive Power of Large Language Models
• Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education
• Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach