一类基于最大效用差的秩相似测度
A Family of Rank Similarity Measures based on Maximized Effectiveness Difference
摘要：
排名相似性度量提供了一种用于量化搜索引擎结果之间的差异而不需要相关性判断的方法。例如，搜索服务的提供者可以使用这些措施来估计所提议的算法改变对大量查询（可能是数百万）的影响，从而识别出影响最大的那些查询。在本文中，我们提出并验证了一个家庭的排名相似性措施，每个来自一个相关的有效性措施。该系列的每个成员都基于在此相关措施下的有效性差异最大化。计算这种最大化的有效性差异（MED）需要解决一个优化问题，不同的难度，这取决于相关的措施。我们提出了几个标准的有效性措施，包括nDCG，MAP和ERR的解决方案。通过实验验证，我们发现MED揭示检索运行之间有意义的差异。从数学上讲，MED是一个度量，无论相关的措施。先前的工作已经建立了一些其他的desiderata排名相似性的搜索的上下文中，我们证明MED满足这些要求。与以前的建议不同，MED允许我们直接从任何已建立的有效性度量中转换有关用户行为的假设，以创建相应的排名相似性度量。此外，MED干净地容纳部分相关性判断，如果完整的相关性信息是可用的，它减少到一个简单的差异有效性值。
英文摘要：
Rank similarity measures provide a method for quantifying differences between search engine results without the need for relevance judgments. For example, the providers of a search service might use such measures to estimate the impact of a proposed algorithmic change across a large number of queries - perhaps millions - identifying those queries where the impact is greatest. In this paper, we propose and validate a family of rank similarity measures, each derived from an associated effectiveness measure. Each member of the family is based on the maximization of effectiveness difference under this associated measure. Computing this maximized effectiveness difference (MED) requires the solution of an optimization problem that varies in difficulty, depending on the associated measure. We present solutions for several standard effectiveness measures, including nDCG, MAP, and ERR. Through an experimental validation, we show that MED reveals meaningful differences between retrieval runs. Mathematically, MED is a metric, regardless of the associated measure. Prior work has established a number of other desiderata for rank similarity in the context of search, and we demonstrate that MED satisfies these requirements. Unlike previous proposals, MED allows us to directly translate assumptions about user behavior from any established effectiveness measure to create a corresponding rank similarity measure. In addition, MED cleanly accommodates partial relevance judgments, and if complete relevance information is available, it reduces to a simple difference between effectiveness values.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
