Instruction‐Tuned Large‐Language Models for Quality Control in Automatic Item Generation: A Feasibility Study

2024/12/19 by Guher Gorgun, Okan Bulut · 1 voice
Computer Science · #Topic Modeling #Intelligent Tutoring Systems and Adaptive Learning #Online Learning and Analytics

paper · doi:10.1111/emip.12663

openalex publication_date 2024/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/22

Abstract

Abstract Automatic item generation may supply many items instantly and efficiently to assessment and learning environments. Yet, the evaluation of item quality persists to be a bottleneck for deploying generated items in learning and assessment settings. In this study, we investigated the utility of using large‐language models, specifically Llama 3‐8B, for evaluating automatically generated cloze items. The trained large‐language model was able to filter out majority of good and bad items accurately. Evaluating items automatically with instruction‐tuned LLMs may aid educators and test developers in understanding the quality of items created in an efficient and scalable manner. The item evaluation process with LLMs may also act as an intermediate step between item creation and field testing to reduce the cost and time associated with multiple rounds of revision.

Citations

• LLaMA: Open and Efficient Foundation Language Models

Discussions

• In a recent article in EM:IP, Gorgun and Bulut explore the use of instruction-tuned large-language models for evaluating the quality of automatically generated items. Learn more: doi.org/10.1111/emip [bsky, 1 points, 0 comments]

Related

• Scaling Item-to-Standard Alignment with Large Language Models: Accuracy, Limits, and Solutions
• How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?
• Calibrating LLM-Based Evaluator
• Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
• Auto prompting without training labels: An LLM cascade for product quality assessment in e-commerce catalogs