2024/02/19 by Sonja Johnson-Yu, Nicholas David Bowman, Johnson-Yu, Sonja +5
Computer Science · Engineering · #Advanced Vision and Imaging #Computers and Society (cs.CY) #FOS: Computer and information sciences #Industrial Vision Systems and Defect Detection #Optical measurement and interference techniques
paper · pdf · doi:10.48550/arxiv.2403.14637
openalex publication_date 2024/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
While the use of programming problems on exams is a common form of summative assessment in CS courses, grading such exam problems can be a difficult and inconsistent process. Through an analysis of historical grading patterns we show that inaccurate and inconsistent grading of free-response programming problems is widespread in CS1 courses. These inconsistencies necessitate the development of methods to ensure more fairer and more accurate grading. In subsequent analysis of this historical exam data we demonstrate that graders are able to more accurately assign a score to a student submission when they have previously seen another submission similar to it. As a result, we hypothesize that we can improve exam grading accuracy by ensuring that each submission that a grader sees is similar to at least one submission they have previously seen. We propose several algorithms for (1) assigning student submissions to graders, and (2) ordering submissions to maximize the probability that a grader has previously seen a similar solution, leveraging distributed representations of student code in order to measure similarity between submissions. Finally, we demonstrate in simulation that these algorithms achieve higher grading accuracy than the current standard random assignment process used for grading.