Exploring the Psychometric Validity of AI-Generated Student Responses: A Study on Virtual Personas' Learning Motivation

2025/11/05 by Wang, Huanxiao
#Artificial Intelligence (cs.AI) #Computers and Society (cs.CY) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2511.07451

Abstract

This study explores whether large language models (LLMs) can simulate valid student responses for educational measurement. Using GPT -4o, 2000 virtual student personas were generated. Each persona completed the Academic Motivation Scale (AMS). Factor analyses(EFA and CFA) and clustering showed GPT -4o reproduced the AMS structure and distinct motivational subgroups.

Citations

• Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis
• Evaluating existing and new validity evidence for the Academic Motivation Scale
• The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education
• Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.

Related

• Leveraging generative artificial intelligence to simulate student learning behavior
• EduPersona: Benchmarking Subjective Ability Boundaries of Virtual Student Agents
• When LLMs Learn to be Students: The SOEI Framework for Modeling and Evaluating Virtual Student Agents in Educational Interaction
• Quantifying Student Success with Generative AI: A Monte Carlo Simulation Informed by Systematic Review
• Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System