比他们的声誉更好。学生相关性评价的信度研究
Better Than Their Reputation? On the Reliability of Relevance Assessments with Students
摘要：
在过去的三年中，我们进行了几个信息检索评价系列与180多名图书情报学的学生谁作出了相关性评估的结果，三个特定的检索服务。在这项研究中，我们不专注于我们的系统的检索性能，但在相关性评估和评估员之间的可靠性。为了量化一致性，我们应用Fleiss的Kappa和Krippendorff的Alpha。当我们比较这两个统计指标时，平均Kappa值为0.37，Alpha值为0.15。我们使用这两个一致性度量来从我们的数据集中删除太不可靠的评估。当计算未过滤和过滤数据集之间的差异时，我们看到均方根误差在0.02和0.12之间。我们认为这是一个明确的指标，不一致影响检索评估的可靠性。我们建议不要使用未经过滤的结果，也不要清楚地记录不一致率。
英文摘要：
During the last three years we conducted several information retrieval evaluation series with more than 180 LIS students who made relevance assessments on the outcomes of three specific retrieval services. In this study we do not focus on the retrieval performance of our system but on the relevance assessments and the inter-assessor reliability. To quantify the agreement we apply Fleiss' Kappa and Krippendorff's Alpha. When we compare these two statistical measures on average Kappa values were 0.37 and Alpha values 0.15. We use the two agreement measures to drop too unreliable assessments from our data set. When computing the differences between the unfiltered and the filtered data set we see a root mean square error between 0.02 and 0.12. We see this as a clear indicator that disagreement affects the reliability of retrieval evaluations. We suggest not to work with unfiltered results or to clearly document the disagreement rates.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
