Analyzing Examinee Comments using DistilBERT and Machine Learning to Ensure Quality Control in Exam Content

2025/04/08 by Ye, Ma
Decision Sciences · Social Sciences · #Academic integrity and plagiarism #Computation and Language (cs.CL) #FOS: Computer and information sciences #Psychometric Methodologies and Testing #Reliability and Agreement in Measurement

paper · pdf · doi:10.48550/arxiv.2504.06465

openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This study explores using Natural Language Processing (NLP) to analyze candidate comments for identifying problematic test items. We developed and validated machine learning models that automatically identify relevant negative feedback, evaluated approaches of incorporating psychometric features enhances model performance, and compared NLP-flagged items with traditionally flagged items. Results demonstrate that candidate feedback provides valuable complementary information to statistical methods, potentially improving test validity while reducing manual review burden. This research offers testing organizations an efficient mechanism to incorporate direct candidate experience into quality assurance processes.

Citations

• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Related

• The Potential of Machine Learning and NLP for Handling Students' Feedback (A Short Survey)
• Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation
• Automatic Detection of Inauthentic Templated Responses in English Language Assessments
• Detecting Problem Statements in Peer Assessments
• Identifying Fairness Issues in Automatically Generated Testing Content