arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.06306cs.IR
收藏

查询性能预测有效性的变化分析

An Analysis of Variations in the Effectiveness of Query Performance Prediction

Debasis Ganguly, Suchana Datta, Mandar Mitra, Derek Greene

摘要:

查询性能预测器估计IR系统对于给定查询的检索有效性。QPP评估的一个重要特征是,由于用于QPP评估的地面真值检索有效性可以用不同的度量来测量,因此地面真值本身不是绝对的,这与其他检索任务(例如ad-hoc检索)相反。出于这一论点,本文的目的是调查如何在地面真理的QPP评估的差异可能会影响QPP实验的结果。我们不仅根据所报告的评估指标的绝对值(例如Pearson的$r$、Kendall的$τ$)来考虑这一点,而且还考虑到当按QPP指标分数排序时不同QPP系统的等级变化。我们的实验表明,观察到的QPP结果可以有很大的不同,无论是在绝对的评估指标值,也在相对系统排名。通过我们的分析,我们报告了QPP评估指标和实验设置的最佳组合,这些组合可能会导致观察结果的较小变化。

英文摘要:

A query performance predictor estimates the retrieval effectiveness of an IR system for a given query. An important characteristic of QPP evaluation is that, since the ground truth retrieval effectiveness for QPP evaluation can be measured with different metrics, the ground truth itself is not absolute, which is in contrast to other retrieval tasks, such as that of ad-hoc retrieval. Motivated by this argument, the objective of this paper is to investigate how such variances in the ground truth for QPP evaluation can affect the outcomes of QPP experiments. We consider this not only in terms of the absolute values of the evaluation metrics being reported (e.g. Pearson's $r$, Kendall's $τ$), but also with respect to the changes in the ranks of different QPP systems when ordered by the QPP metric scores. Our experiments reveal that the observed QPP outcomes can vary considerably, both in terms of the absolute evaluation metric values and also in terms of the relative system ranks. Through our analysis, we report the optimal combinations of QPP evaluation metric and experimental settings that are likely to lead to smaller variations in the observed results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑