本文提出 GPRP 与 Full Stage Learning to Rank 统一框架,通过估计并适配下游阶段选择偏差优化多阶段推荐排序,并在仿真和短视频平台在线 A/B 测试中验证其显著收益。
摘要:
概率排序原则(Probability Ranking Principle,PRP)一直被视为信息检索(IR)系统设计中的基础性标准。该原则要求 IR 模块返回的结果列表按照潜在的用户兴趣进行排序,从而最大化结果的效用。\n然而,我们指出,在当代 IR 系统的每个阶段中不加区分地应用 PRP 是不合适的。此类系统包含多个阶段(例如本文考察的召回、预排序、排序和重排序阶段)。每个阶段模型中固有的选择偏差会显著影响最终展示给用户的结果。\n为解决这一问题,我们针对多阶段系统提出了一种改进的排序原则,即广义概率排序原则(Generalized Probability Ranking Principle,GPRP),以同时强调系统流水线中每个阶段的选择偏差以及用户的潜在兴趣。\n我们通过一个名为 Full Stage Learning to Rank 的统一算法框架实现 GPRP。我们的核心思想是,首先估计后续阶段中的选择偏差,然后学习一个最符合下游模块选择偏差的排序模型,从而将其排名最高的结果送入系统输出的最终排序列表。\n我们使用仿真实验和在一家领先短视频推荐平台上开展的在线 A/B 测试,对所开发的 Full Stage Learning to Rank 解决方案进行了广泛实验评估。实验证明,该算法在召回和排序阶段均有效。自部署以来,该算法已为平台带来持续且显著的性能提升。
英文摘要:
The Probability Ranking Principle (PRP) has been considered as the foundational standard in the design of information retrieval (IR) systems. The principle requires an IR module's returned list of results to be ranked with respect to the underlying user interests, so as to maximize the results' utility.
Nevertheless, we point out that it is inappropriate to indiscriminately apply PRP through every stage of a contemporary IR system. Such systems contain multiple stages (e.g., retrieval, pre-ranking, ranking, and re-ranking stages, as examined in this paper). The \emph{selection bias} inherent in the model of each stage significantly influences the results that are ultimately presented to users.
To address this issue, we propose an improved ranking principle for multi-stage systems, namely the Generalized Probability Ranking Principle (GPRP), to emphasize both the selection bias in each stage of the system pipeline as well as the underlying interest of users.
We realize GPRP via a unified algorithmic framework named Full Stage Learning to Rank. Our core idea is to first estimate the selection bias in the subsequent stages and then learn a ranking model that best complies with the downstream modules' selection bias so as to deliver its top ranked results to the final ranked list in the system's output.
We performed extensive experiment evaluations of our developed Full Stage Learning to Rank solution, using both simulations and online A/B tests in one of the leading short-video recommendation platforms. The algorithm is proved to be effective in both retrieval and ranking stages. Since deployed, the algorithm has brought consistent and significant performance gain to the platform.
Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking
University of Waterloo(滑铁卢大学)
在信息检索(IR)中,候选集修剪通常用于加速两阶段相关性排序。然而,这样的方法缺乏精确的误差控制,并且经常以经验的方式权衡准确性与计算效率,缺乏理论保证。本文提出了相关性排序候选集剪枝的认证误差控制概念,即保证剪枝后的测试误差以高概率控制在用户指定的阈值之下。域内和域外的实验表明,我们的方法成功地修剪第一阶段检索到的候选集,以提高第二阶段的重新排序速度,同时满足预先指定的精度约束,在这两个设置。例如,在MS MARCO Passage v1上,我们的方法产生的平均候选集大小为27/1000,这将重新排序速度提高了约37倍,而MRR@10大于预先指定的值0.38,经验覆盖率约为90%,经验基线无法提供这种保证。代码和数据可在https://github.com/alexlimh/CEC-Ranking上获得。
PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking
Ping An Health Technology, Beijing, China Peking University, Beijing, China Ping An Property \& Casualty Insurance Company, Shenzhen, China Central University of Finance;Economics, Beijing, China Ping An Health Cloud Company Limited., Shenzhen, China Ping An International Smart City Technology Co., Ltd., Shenzhen, China
Tandon School of Engineering, New York University(纽约大学坦登工程学院);David R. Cheriton School of Computer Science, University of Waterloo(滑铁卢大学大卫·R·切里顿计算机科学学院);Courant Institute of Mathematical Sciences, New York University(纽约大学柯朗数学科学研究所);Center for Data Science, New York University(纽约大学数据科学中心);Facebook AI Research(脸书人工智能研究院)
Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation
University of Western Ontario(西安大略大学);Vector Institute for Artificial Intelligence(矢量人工智能研究所);Dalhousie University(达尔豪斯大学);University of Toronto(多伦多大学)
Ensemble Ranking Model with Multiple Pretraining Strategies for Web Search
一个有效的排序模型通常需要大量的训练数据来学习文档和查询之间的相关性。用户点击经常被用作训练数据,因为它们可以指示相关性并且收集起来很便宜,但是它们包含大量的偏差和噪声。在减轻模拟用户点击中的各种类型的偏差以基于多个特征训练有效的学习排名模型方面已经有了一些工作。然而,如何在具有真实点击数据的大规模预训练模型上有效地使用这些方法是未知的。为了减轻现实世界中的数据偏差,我们在预训练阶段引入了基于统计学的特征,改进了排名目标,添加了随机否定,并校准了倾向计算。然后,我们微调几个预训练的模型,并训练一个集成模型,以在微调阶段将来自各种预训练模型的所有预测与人类注释数据聚合在一起。我们的方法在WSDM Cup 2023的“网络搜索预训练”任务中获得第三名,比第四名的团队好22.6%。
Multi-stage Ensemble Model for Cross-market Recommendation
Politecnico di Milano(米兰理工大学)
本文介绍了我们的团队PolimiRank为WSDM Cup 2022提供的跨市场推荐解决方案。比赛的目标是有效地利用从不同市场提取的信息,以提高两个目标市场的推荐排名准确性。我们的模型是基于属于不同市场的数据组合的多阶段方法。在第一阶段,国家的最先进的预测器被用来预测得分的用户项夫妇,这是在以下2个阶段,采用一个简单的线性组合和更强大的梯度提升决策树技术。我们队在决赛中名列第四。