arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07121cs.IRstat.ME
收藏

Rank-Biased Overlap 中并列项的处理

The Treatment of Ties in Rank-Biased Overlap

Matteo Corsi, Julián Urbano

AI总结:

本文针对 Rank-Biased Overlap(RBO)度量中并列项处理不完善的问题,提出广义公式以支持前缀计算,并推导出有参考排序和无参考排序两种统计学风格变体,通过合成与 TREC 数据验证了并列项感知度量与原始度量存在显著差异。

摘要:

Rank-Biased Overlap(RBO)是一种用于不定长排序的相似度度量:它具有顶部加权特性,并且当只知道排序的前缀或两个排序只有部分共同项时也可以计算。它被广泛用于例如通过比较搜索引擎对相同查询所检索文档的排序来分析搜索引擎之间的差异。然而在这些场景中,经常会出现得分相同的并列文档。遗憾的是,RBO 中对并列项的处理仍然较为肤浅且不完整,具体表现为尚不清楚如何仅从排序前缀来计算它。此外,现有的并列项处理方式与统计学领域传统采用的方式有很大不同,尤其是在 Kendall 和 Spearman 等秩相关系数中所见的处理方式。在本文中,我们提出了一种用于处理并列项的 RBO 广义公式,借此我们通过展示如何进行前缀评估来补全原始定义。我们还利用该公式完整推导出两种变体,它们与统计学文献中的处理方式一致:一种是在存在参考排序可供比较时使用,另一种是在不存在参考排序时使用。总体而言,这三种变体通过明确确定并列项的含义以及应如何处理它们,为研究人员在使用 RBO 比较排序时提供了灵活性。最后,我们使用合成数据和 TREC 数据展示了这些新的并列项感知 RBO 度量的用法。我们表明,其得分可能与原始的并列项不感知 RBO 度量存在显著差异,在原始度量中并列项必须通过随机方式或按文档 ID 等任意标准来打破并列。总体而言,这些结果证明了在 RBO 等排序相似度度量中正确考虑并列项的必要性。

英文摘要:

Rank-Biased Overlap (RBO) is a similarity measure for indefinite rankings: it is top-weighted, and can be computed when only a prefix of the rankings is known or when they have only some items in common. It is widely used for instance to analyze differences between search engines by comparing the rankings of documents they retrieve for the same queries. In these situations, though, it is very frequent to find tied documents that have the same score. Unfortunately, the treatment of ties in RBO remains superficial and incomplete, in the sense that it is not clear how to calculate it from the ranking prefixes only. In addition, the existing way of dealing with ties is very different from the one traditionally followed in the field of Statistics, most notably found in rank correlation coefficients such as Kendall's and Spearman's. In this paper we propose a generalized formulation for RBO to handle ties, thanks to which we complete the original definitions by showing how to perform prefix evaluation. We also use it to fully develop two variants that align with the ones found in the Statistics literature: one when there is a reference ranking to compare to, and one when there is not. Overall, these three variants provide researchers with flexibility when comparing rankings with RBO, by clearly determining what ties mean, and how they should be treated. Finally, using both synthetic and TREC data, we demonstrate the use of these new tie-aware RBO measures. We show that the scores may differ substantially from the original tie-unaware RBO measure, where ties had to be broken at random or by arbitrary criteria such as by document ID. Overall, these results evidence the need for a proper account of ties in rank similarity measures such as RBO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑