NDCG型排序测度的理论分析
A Theoretical Analysis of NDCG Type Ranking Measures
摘要：
排序中的一个核心问题是设计一个用于评价排序函数的排序度量。在本文中，我们研究，从理论的角度来看，广泛使用的归一化贴现累积增益（NDCG）型排名措施。虽然有大量的NDCG的实证研究，很少有人知道它的理论特性。我们首先表明，无论排名功能是，标准的NDCG采用对数折扣，收敛到1的项目数量排名走向无穷大。乍一看，这个结果非常令人惊讶。这似乎意味着，NDCG不能区分好的和坏的排名功能，矛盾的NDCG在许多应用中的经验成功。为了有一个更深入的了解一般的排名措施，我们提出了一个概念，称为一致的可扩展性。这个概念捕捉了一种直觉，即排名度量应该具有这样的属性：对于每一对实质上不同的排名函数，排名度量可以在几乎所有数据集上以一致的方式决定哪一个更好。我们表明，NDCG与对数折扣具有一致的可扩展性，虽然它收敛到相同的限制，所有的排名功能。接下来，我们根据一致性的概念来描述NDCG的所有可行的折扣函数的集合。具体来说，我们表明，NDCG是否具有一致的可验证性取决于折扣衰减的速度，1/r是一个临界点。然后，我们转向NDCG的截断版本，即，NDCG@k。我们分析了NDCG@k对于k和折扣函数的各种选择的可扩展性。在真实Web搜索数据集上的实验结果与理论一致。
英文摘要：
A central problem in ranking is to design a ranking measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the widely used Normalized Discounted Cumulative Gain (NDCG)-type ranking measures. Although there are extensive empirical studies of NDCG, little is known about its theoretical properties. We first show that, whatever the ranking function is, the standard NDCG which adopts a logarithmic discount, converges to 1 as the number of items to rank goes to infinity. On the first sight, this result is very surprising. It seems to imply that NDCG cannot differentiate good and bad ranking functions, contradicting to the empirical success of NDCG in many applications. In order to have a deeper understanding of ranking measures in general, we propose a notion referred to as consistent distinguishability. This notion captures the intuition that a ranking measure should have such a property: For every pair of substantially different ranking functions, the ranking measure can decide which one is better in a consistent manner on almost all datasets. We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions. We next characterize the set of all feasible discount functions for NDCG according to the concept of consistent distinguishability. Specifically we show that whether NDCG has consistent distinguishability depends on how fast the discount decays, and 1/r is a critical point. We then turn to the cut-off version of NDCG, i.e., NDCG@k. We analyze the distinguishability of NDCG@k for various choices of k and the discount functions. Experimental results on real Web search datasets agree well with the theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
