利用人类反馈规模化教育数据集：结合众包工作者与比较判断
Leveraging Human Feedback to Scale Educational Datasets: Combining Crowdworkers and Comparative Judgement
AI总结：
针对教育数据标注难规模化问题，研究通过实验对比发现，结合非专业众包工作者与比较判断法可显著提升评分者间信度，为规模化教育数据集提供了新的有效路径。
摘要：
机器学习模型在教育场景中有诸多潜在有益应用，但其发展的一大关键障碍是获取足够数据来训练这些模型。传统上，教育数据标注依赖高技能评分者使用复杂的多类别评分标准，这一过程成本高昂且难以规模化。另一种更具可扩展性的替代方案是使用非专业众包工作者评估学生作业，然而，在使用非专业工作者时维持足够高的准确率和评分者间信度颇具挑战。本文报告了两项实验，探究使用非专业众包工作者与比较判断法评估复杂学生数据的效果。研究雇佣众包工作者评估学生对开放式阅读理解问题的作答。众包工作者被随机分配至两种条件之一：对照组需判断答案正确或错误（即分类判断），处理组看到相同的问题与答案，但需判断两个候选答案中哪一个更正确（即比较/基于偏好的判断）。研究发现，使用比较判断法显著提升了两项任务的评分者间信度。这些结果与教育评估领域中关于比较判断优势的既有成熟文献一致，也契合人工智能研究的近期趋势——在与非专业众包工作者合作时，比较判断正成为为模型输出提供人类反馈的首选方法。但据我们所知，这些结果首次且重要地证明了结合比较判断与众包工作者评估教育数据的有益效果。
英文摘要：
Machine Learning models have many potentially beneficial applications in education settings, but a key barrier to their development is securing enough data to train these models. Labelling educational data has traditionally relied on highly skilled raters using complex, multi-class rubrics, making the process expensive and difficult to scale. An alternative, more scalable approach could be to use non-expert crowdworkers to evaluate student work, however, maintaining sufficiently high levels of accuracy and inter-rater reliability when using non-expert workers is challenging. This paper reports on two experiments investigating using non-expert crowdworkers and comparative judgement to evaluate complex student data. Crowdworkers were hired to evaluate student responses to open-ended reading comprehension questions. Crowdworkers were randomly assigned to one of two conditions: the control, where they were asked to decide whether answers were correct or incorrect (i.e., a categorical judgement), or the treatment, where they were shown the same question and answers, but were instead asked to decide which of two candidate answers was more correct (i.e., a comparative/preference-based judgement). We found that using comparative judgement substantially improved inter-rater reliability on both tasks. These results are in-line with well-established literature on the benefits of comparative judgement in the field of educational assessment, as well as with recent trends in artificial intelligence research, where comparative judgement is becoming the preferred method for providing human feedback on model outputs when working with non-expert crowdworkers. However, to our knowledge, these results are novel and important in demonstrating the beneficial effects of using the combination of comparative judgement and crowdworkers to evaluate educational data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
