arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.14637cs.CY
收藏

SimGrade:使用代码相似性度量实现更准确的人工评分

SimGrade: Using Code Similarity Measures for More Accurate Human Grading

Sonja Johnson-Yu, Nicholas Bowman, Mehran Sahami, Chris Piech

AI总结:

针对CS1课程编程题人工评分不一致问题,提出SimGrade方法,利用代码分布式表示衡量相似性并设计分配与排序算法,使评分者优先接触相似提交,显著提升了评分准确性。

摘要:

虽然在考试中使用编程题是计算机科学课程中总结性评估的常见形式,但对这类考试题进行评分可能是一个困难且不一致的过程。通过对历史评分模式的分析,我们表明,在CS1课程中,对自由回答编程题的不准确和不一致评分是普遍存在的。这些不一致性使得有必要开发方法来确保更公平、更准确的评分。在对这一历史考试数据的后续分析中,我们证明,当评分者之前看过另一个与其相似的提交时,他们能够更准确地为学生的提交分配分数。因此,我们假设,通过确保评分者看到的每份提交都与他们之前看过的至少一份提交相似,我们可以提高考试评分的准确性。我们提出了几种算法,用于(1)将学生提交分配给评分者,以及(2)对提交进行排序,以最大化评分者之前看过相似解决方案的概率,这些算法利用学生代码的分布式表示来衡量提交之间的相似性。最后,我们在仿真中证明,这些算法比当前用于评分的标准随机分配过程实现了更高的评分准确性。

英文摘要:

While the use of programming problems on exams is a common form of summative assessment in CS courses, grading such exam problems can be a difficult and inconsistent process. Through an analysis of historical grading patterns we show that inaccurate and inconsistent grading of free-response programming problems is widespread in CS1 courses. These inconsistencies necessitate the development of methods to ensure more fairer and more accurate grading. In subsequent analysis of this historical exam data we demonstrate that graders are able to more accurately assign a score to a student submission when they have previously seen another submission similar to it. As a result, we hypothesize that we can improve exam grading accuracy by ensuring that each submission that a grader sees is similar to at least one submission they have previously seen. We propose several algorithms for (1) assigning student submissions to graders, and (2) ordering submissions to maximize the probability that a grader has previously seen a similar solution, leveraging distributed representations of student code in order to measure similarity between submissions. Finally, we demonstrate in simulation that these algorithms achieve higher grading accuracy than the current standard random assignment process used for grading.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑