arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.07908cs.IR
收藏

最佳情况检索评估:利用字典精度提升倒数排名的敏感性

Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision

Fernando Diaz

AI总结:

针对倒数排名指标区分系统敏感性不足的问题,研究结合最佳情况检索概念推广该指标,提出新的字典精度评估指标,在保留原指标检测差异能力的同时提升了检索与推荐任务评估的敏感性和鲁棒性。

摘要:

在各类排序任务中,研究人员使用倒数排名(reciprocal rank)来衡量仅关注1个相关项的用户场景下的系统效果。尽管该指标应用广泛,已有证据表明倒数排名在区分不同系统时表现脆弱,这一缺陷在现代评估场景中更为突出——当前的高精度系统往往难以被有效区分。针对倒数排名敏感性不足的问题,我们将其与最佳情况检索(best-case retrieval)概念关联并引入该概念,这是一种聚焦于在不同召回需求下,评估排序结果对最满意用户的质量的评估方法。基于该视角,我们推广了倒数排名,并定义了一种新的基于偏好的评估指标,称为字典精度(lexicographic precision,简称lexiprecision)。通过数学构造,我们确保lexiprecision保留倒数排名能检测到的差异,同时在大量检索和推荐任务中,经实证验证提升了敏感性和鲁棒性。

英文摘要:

Across a variety of ranking tasks, researchers use reciprocal rank to measure the effectiveness for users interested in exactly one relevant item. Despite its widespread use, evidence suggests that reciprocal rank is brittle when discriminating between systems. This brittleness, in turn, is compounded in modern evaluation settings where current, high-precision systems may be difficult to distinguish. We address the lack of sensitivity of reciprocal rank by introducing and connecting it to the concept of best-case retrieval, an evaluation method focusing on assessing the quality of a ranking for the most satisfied possible user across possible recall requirements. This perspective allows us to generalize reciprocal rank and define a new preference-based evaluation we call lexicographic precision or lexiprecision. By mathematical construction, we ensure that lexiprecision preserves differences detected by reciprocal rank, while empirically improving sensitivity and robustness across a broad set of retrieval and recommendation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑