arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.09340cs.IR
收藏

你的Qrels判别力有多强?如何研究文档 adjudication 方法的统计显著性

How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods

David Otero, Javier Parapar, Nicola Ferro

AI总结:

针对传统文档adjudication方法评估忽略统计显著性差异的问题,提出一种评估低成本方法保留系统成对显著性差异能力的新方法,发现排名相关性最优的方法未必能保留统计显著性。

摘要:

构建用于离线检索评估的测试集需要人工判断文档的相关性,这项成本高昂的工作推动了大量以更低评估成本构建基准的方法研究。在这一领域,adjudication( adjudication )方法会主动确定专家需要评审的文档及其评审顺序,以更充分地利用评估预算或降低预算成本。研究人员通过衡量完整测试集下已知的系统金标排名与低成本测试集下观测到的系统排名之间的相关性,来评估这些方法的质量。这种传统分析忽略了低成本判断是否以及如何影响系统间相对于完整测试集的统计显著性差异。我们通过提出一种新方法填补了这一空白,该方法用于评估低成本adjudication方法如何像完整测试集一样保留系统间的成对显著性差异。换言之,传统方法寻求的是回答“系统A是否优于系统B”这一问题的稳定性,而我们提出的方法寻求的是回答“系统A是否显著优于系统B”这一问题的稳定性,这是研究人员保证其结果普适性需要回答的核心问题。在诸多结果中,我们发现,在系统排名相关性方面表现最佳的方法,并不总是与能保留统计显著性的方法一致。

英文摘要:

Creating test collections for offline retrieval evaluation requires human effort to judge documents' relevance. This expensive activity motivated much work in developing methods for constructing benchmarks with fewer assessment costs. In this respect, adjudication methods actively decide both which documents and the order in which experts review them, in order to better exploit the assessment budget or to lower it. Researchers evaluate the quality of those methods by measuring the correlation between the known gold ranking of systems under the full collection and the observed ranking of systems under the lower-cost one. This traditional analysis ignores whether and how the low-cost judgements impact on the statistically significant differences among systems with respect to the full collection. We fill this void by proposing a novel methodology to evaluate how the low-cost adjudication methods preserve the pairwise significant differences between systems as the full collection. In other terms, while traditional approaches look for stability in answering the question "is system A better than system B?", our proposed approach looks for stability in answering the question "is system A significantly better than system B?", which is the ultimate questions researchers need to answer to guarantee the generalisability of their results. Among other results, we found that the best methods in terms of ranking of systems correlation do not always match those preserving statistical significance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑