IR评估的联合上下限归一化
Joint Upper & Lower Bound Normalization for IR Evaluation
摘要：
在本文中，我们提出了一个新的角度对IR评估提出了一个新的家庭的评估指标，其中现有的流行指标（例如，nDCG、MAP）通过引入查询特定的下限（LB）规范化项来定制。虽然原始的nDCG、MAP等度量基于理想的排名列表在其上界方面被归一化，但是尚未研究针对它们的对应的LB归一化。具体来说，我们引入了两个不同的变种，建议LB规范化，其中的下限估计从一个随机排名的相应文件中存在的评估集。接下来，我们进行了两个案例研究，通过实例化两个流行的IR评估指标的新框架（具有两个变体，例如，DCG_UL_V1，2和MSP_UL_V1，2），然后与没有所提出的LB归一化的传统度量进行比较。在两个不同的数据集上使用八种LETOR方法进行实验，结果表明新的LB归一化度量具有以下性质：1）在上下界（UL）归一化版本中，原始度量的统计显著性差异不再保持统计显著性，反之亦然，特别是对于无信息的查询集。2)与原始度量相比，我们提出的UL归一化度量在不同的数据集上表现出更高的区分能力和更好的一致性。这些研究结果表明，IR社区应该考虑UL归一化计算时，nDCG和MAP和更深入的研究UL归一化一般IR评估是必要的。
英文摘要：
In this paper, we present a novel perspective towards IR evaluation by proposing a new family of evaluation metrics where the existing popular metrics (e.g., nDCG, MAP) are customized by introducing a query-specific lower-bound (LB) normalization term. While original nDCG, MAP etc. metrics are normalized in terms of their upper bounds based on an ideal ranked list, a corresponding LB normalization for them has not yet been studied. Specifically, we introduce two different variants of the proposed LB normalization, where the lower bound is estimated from a randomized ranking of the corresponding documents present in the evaluation set. We next conducted two case-studies by instantiating the new framework for two popular IR evaluation metric (with two variants, e.g., DCG_UL_V1,2 and MSP_UL_V1,2 ) and then comparing against the traditional metric without the proposed LB normalization. Experiments on two different data-sets with eight Learning-to-Rank (LETOR) methods demonstrate the following properties of the new LB normalized metric: 1) Statistically significant differences (between two methods) in terms of original metric no longer remain statistically significant in terms of Upper Lower (UL) Bound normalized version and vice-versa, especially for uninformative query-sets. 2) When compared against the original metric, our proposed UL normalized metrics demonstrate higher Discriminatory Power and better Consistency across different data-sets. These findings suggest that the IR community should consider UL normalization seriously when computing nDCG and MAP and more in-depth study of UL normalization for general IR evaluation is warranted.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
