arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1404.3656cs.LGcs.IR
收藏

顺序同侪评分方法

Methods for Ordinal Peer Grading

Cornell University(康奈尔大学)

Karthik Raman, Thorsten Joachims

摘要:

MOOC有可能以其广泛的覆盖面和可访问性彻底改变高等教育,但它们要求教师提出可扩展的替代传统学生评估的方法。同伴评分--让学生们互相评价--是解决大规模评价问题的一种很有前途的方法,因为“评分者”的数量自然会随着学生的数量而变化。然而,学生没有受过评分训练,这意味着人们不能期望与传统环境中相同的评分技能水平。利用序数反馈比基数反馈更容易提供且更可靠的广泛证据,因此期望允许同龄评分者做出序数陈述(例如,“项目X比项目Y更好”),而不要求他们做出基数陈述(例如,“项目X是B-”)。因此,在本文中,我们研究的问题,自动推断学生成绩从序数同行反馈,而不是现有的方法,需要基数同行反馈。我们制定了顺序同行评分问题的等级聚合问题的一种类型,并探讨了几种概率模型下,估计学生成绩和评分员的可靠性。我们研究了这些方法的适用性,使用从一个真实的类-教师和TA等级作为基线-收集的同行评分数据,并证明了序数反馈技术的有效性相比,现有的基数同行评分方法。最后,我们将这些同行评分技术与传统的评价技术进行了比较。

英文摘要:

MOOCs have the potential to revolutionize higher education with their wide outreach and accessibility, but they require instructors to come up with scalable alternates to traditional student evaluation. Peer grading -- having students assess each other -- is a promising approach to tackling the problem of evaluation at scale, since the number of "graders" naturally scales with the number of students. However, students are not trained in grading, which means that one cannot expect the same level of grading skills as in traditional settings. Drawing on broad evidence that ordinal feedback is easier to provide and more reliable than cardinal feedback, it is therefore desirable to allow peer graders to make ordinal statements (e.g. "project X is better than project Y") and not require them to make cardinal statements (e.g. "project X is a B-"). Thus, in this paper we study the problem of automatically inferring student grades from ordinal peer feedback, as opposed to existing methods that require cardinal peer feedback. We formulate the ordinal peer grading problem as a type of rank aggregation problem, and explore several probabilistic models under which to estimate student grades and grader reliability. We study the applicability of these methods using peer grading data collected from a real class -- with instructor and TA grades as a baseline -- and demonstrate the efficacy of ordinal feedback techniques in comparison to existing cardinal peer grading methods. Finally, we compare these peer-grading techniques to traditional evaluation techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑