简单序数同侪评分的有效性如何？
How effective can simple ordinal peer grading be?
摘要：
有序同行评分已被提出作为一个简单的和可扩展的解决方案，计算可靠的信息，学生的表现在大规模的开放式在线课程。这个想法是外包给学生自己的评分任务如下。考试结束后，每个学生都被要求根据质量对同学的一堆试卷进行排名。然后，聚合规则将各个排名组合成包含所有学生的全局排名。我们定义了一个广泛的类简单的聚合规则，我们称之为类型排序聚合规则，并提出了一个理论框架来评估其有效性。当关于学生的评分行为的统计信息可用时（根据表征来自学生群体的平均学生的评分行为的噪声矩阵），框架可以用于计算关于一系列性能目标的来自该类的最优规则，所述性能目标将由聚合规则返回的排名与底层的真实排名进行比较。例如，当学生正确评分时，被称为Borda的自然规则被证明是最优的。此外，我们提出了广泛的模拟，验证我们的理论，并证明它是非常准确的预测性能的聚合规则，即使只有粗略的信息分级行为（即，噪声矩阵的近似）是可用的。无论是在我们的理论框架的应用，并在我们的模拟，我们利用数据的学生，已被提取的两个实地实验在帕特雷大学的分级行为。
英文摘要：
Ordinal peer grading has been proposed as a simple and scalable solution for computing reliable information about student performance in massive open online courses. The idea is to outsource the grading task to the students themselves as follows. After the end of an exam, each student is asked to rank -- in terms of quality -- a bundle of exam papers by fellow students. An aggregation rule then combines the individual rankings into a global one that contains all students. We define a broad class of simple aggregation rules, which we call type-ordering aggregation rules, and present a theoretical framework for assessing their effectiveness. When statistical information about the grading behaviour of students is available (in terms of a noise matrix that characterizes the grading behaviour of the average student from a student population), the framework can be used to compute the optimal rule from this class with respect to a series of performance objectives that compare the ranking returned by the aggregation rule to the underlying ground truth ranking. For example, a natural rule known as Borda is proved to be optimal when students grade correctly. In addition, we present extensive simulations that validate our theory and prove it to be extremely accurate in predicting the performance of aggregation rules even when only rough information about grading behaviour (i.e., an approximation of the noise matrix) is available. Both in the application of our theoretical framework and in our simulations, we exploit data about grading behaviour of students that have been extracted from two field experiments in the University of Patras.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
