通过顺序置信建模提升教育评估中标记评分的准确性
Enhancing Marker Scoring Accuracy through Ordinal Confidence Modelling in Educational Assessments
AI总结：
本研究通过引入核加权有序分类交叉熵损失函数，提升教育评估中自动评分的置信度建模，实现更高的CEFR一致性评分。
摘要：
在自动作文评分（AES）中，一个关键的伦理挑战是确保只有在满足高可靠性标准时才释放评分。置信建模通过为每个自动评分分配可靠性估计措施，以置信分数的形式来解决这一问题。在本研究中，我们将置信度估计视为一个分类任务：预测AES生成的评分是否正确地将考生置于适当的CEFR等级。虽然这是一个二元决策，但我们利用评分领域的内在粒度以两种方式加以利用。首先，我们将任务重新表述为一个n元分类问题，使用分数分箱。其次，我们引入了一组新的核加权有序分类交叉熵（KWOCCE）损失函数，这些函数纳入了CEFR标签的有序结构。我们表现最好的模型实现了F1得分为0.97，并使系统能够释放47%的评分，其CEFR一致率为100%，99%的评分至少有95%的CEFR一致率——与独立的AES模型相比，该模型释放所有AM预测评分时，CEFR一致率为约92%（近似值）。\n
英文摘要：
A key ethical challenge in Automated Essay Scoring (AES) is ensuring that scores are only released when they meet high reliability standards. Confidence modelling addresses this by assigning a reliability estimate measure, in the form of a confidence score, to each automated score. In this study, we frame confidence estimation as a classification task: predicting whether an AES-generated score correctly places a candidate in the appropriate CEFR level. While this is a binary decision, we leverage the inherent granularity of the scoring domain in two ways. First, we reformulate the task as an n-ary classification problem using score binning. Second, we introduce a set of novel Kernel Weighted Ordinal Categorical Cross Entropy (KWOCCE) loss functions that incorporate the ordinal structure of CEFR labels. Our best-performing model achieves an F1 score of 0.97, and enables the system to release 47% of scores with 100% CEFR agreement and 99% with at least 95% CEFR agreement -compared to approximately 92% (approx.) CEFR agreement from the standalone AES model where we release all AM predicted scores.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
