arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.11370cs.IR
收藏

召回率、鲁棒性与字典序评估

Recall, Robustness, and Lexicographic Evaluation

Fernando Diaz, Michael D. Ekstrand, Bhaskar Mitra

AI总结:

针对召回率在排名评估中应用缺乏正式动机的问题,本文从召回导向性、鲁棒性和字典序评估三方面分析,定义召回导向性并开发lexirecall方法,实证表明其收敛效度与更高敏感性,深化召回率理解并推动其应用。

摘要:

尽管召回率最初用于评估项目集合,但如今常被用于评估项目排名,包括推荐系统、检索系统及其他机器学习系统生成的排名。在缺乏正式评估动机的情况下应用召回率,导致其被批评为模糊或不恰当的指标。鉴于此争议,我们从正式视角反思排名中召回率的测量问题。我们的分析包含三个原则:召回率、鲁棒性和字典序评估。首先,我们正式定义“召回导向性”为指标对希望找到所有相关项目的用户的敏感性。其次,我们从对可能的内容消费者和提供者的鲁棒性视角分析召回导向性,将召回率与近期关于公平排名的讨论联系起来。最后,我们通过开发基于字典序比较的实用偏好评估方法,扩展对召回率的概念和理论处理。通过在三个推荐任务和17个信息检索任务上的广泛实证分析,我们证实新评估方法lexirecall具有收敛效度(即与现有召回指标相关),且在区分能力和缺失标签存在时的稳定性方面表现出显著更高的敏感性。我们的概念、理论和实证分析显著深化了对召回率的理解,并通过与鲁棒性和公平性的联系推动其应用。

英文摘要:

Although originally developed to evaluate sets of items, recall is often used to evaluate rankings of items, including those produced by recommender, retrieval, and other machine learning systems. The application of recall without a formal evaluative motivation has led to criticism of recall as a vague or inappropriate measure. In light of this debate, we reflect on the measurement of recall in rankings from a formal perspective. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation. First, we formally define `recall-orientation' as the sensitivity of a metric to a user interested in finding every relevant item. Second, we analyze recall-orientation from the perspective of robustness with respect to possible content consumers and providers, connecting recall to recent conversations about fair ranking. Finally, we extend this conceptual and theoretical treatment of recall by developing a practical preference-based evaluation method based on lexicographic comparison. Through extensive empirical analysis across three recommendation tasks and 17 information retrieval tasks, we establish that our new evaluation method, lexirecall, has convergent validity (i.e., it is correlated with existing recall metrics) and exhibits substantially higher sensitivity in terms of discriminative power and stability in the presence of missing labels. Our conceptual, theoretical, and empirical analysis substantially deepens our understanding of recall and motivates its adoption through connections to robustness and fairness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑