Exploring the potential of using an AI language model for automated essay scoring

2023/04/19 by Atsushi Mizumoto, Masaki Eguchi · 1 voice · 1 citation
Computer Science · Medicine · #Text Readability and Simplification #Topic Modeling #Artificial Intelligence in Healthcare and Education

paper · doi:10.1016/j.rmal.2023.100050

openalex publication_date 2023/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

The widespread adoption of ChatGPT, an AI language model, has the potential to bring about significant changes to the research, teaching, and learning of foreign languages. The present study aims to leverage this technology to perform automated essay scoring (AES) and evaluate its reliability and accuracy. Specifically, we utilized the GPT-3 text-davinci-003 model to automatically score all 12,100 essays contained in the ETS Corpus of Non-Native Written English (TOEFL11) and compared these scores to benchmark levels. The study also explored the extent to which linguistic features influence AES with GPT. The results showed that AES using GPT has a certain level of accuracy and reliability and could provide valuable support for human evaluations. Furthermore, the analysis revealed that utilizing linguistic features could enhance the accuracy of the scoring. These findings suggest that AI language models, such as ChatGPT, can be effectively utilized as AES tools, potentially revolutionizing methods of writing evaluation and feedback in both research and practice. The paper concludes by discussing the practical implications of using GPT for AES and exploring prospective future considerations.

Citations

• A statistical model for the analysis of ordinal level dependent variables
• How Big Is “Big”? Interpreting Effect Sizes in L2 Research
• The Measurement of Observer Agreement for Categorical Data
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• R-squared for Bayesian Regression Models
• Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion)
• Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study
• Calculating the Relative Importance of Multiple Regression Predictor Variables Using Dominance Analysis and Random Forests

Cited by

• Descriptive writing using generative AI as a cognitive scaffold in the metaverse environment: university students’ perceptions, learning engagement, and performance

Discussions

• Exploring the potential of using an AI language model for automated essay scoring. doi.org/10.1016/j.rm... "The results showed that AES (automated essay scoring) using GPT has a certain level of acc [bsky, 0 points, 0 comments]