分而治之：从多任务的角度研究推荐系统中更好的嵌入式检索
Divide and Conquer: Towards Better Embedding-based Retrieval for Recommender Systems From a Multi-task Perspective
摘要：
基于嵌入的检索（EBR）方法由于其简单有效而被广泛应用于现代推荐系统中。然而，在生产环境中部署和迭代EBR的过程中，我们仍然发现了现有方法中的一些基本问题。首先，当处理大量的候选项目时，EBR模型通常难以平衡区分高度相关的项目（正面）与不相关的项目（容易的负面）以及与一些相关但不竞争的项目（硬负面）的性能。此外，由于最近向量搜索的“贪婪”性质，我们对检索结果的多样性和公平性几乎没有控制。这些问题损害了EBR方法在大规模工业场景中的性能。本文介绍了一种简单且经过生产验证的解决方案来克服这些问题。所提出的解决方案采取分而治之的方法：整个集合的候选项被分成多个集群，我们运行EBR检索相关的候选人从每个集群并行，然后从每个集群的顶部候选人结合一些可控的合并策略。这种方法允许我们的EBR模型只专注于从大多数硬阴性中区分阳性。它还可以从多任务学习（MTL）的角度进行进一步改进：每个集群内的检索问题可以被视为单独的任务;受到最近在提示和前缀调整方面取得的成功的启发，我们提出了一种有效的任务自适应技术，进一步提高了每个集群内的检索性能，其开销可以忽略不计。
英文摘要：
Embedding-based retrieval (EBR) methods are widely used in modern recommender systems thanks to its simplicity and effectiveness. However, along the journey of deploying and iterating on EBR in production, we still identify some fundamental issues in existing methods. First, when dealing with large corpus of candidate items, EBR models often have difficulties in balancing the performance on distinguishing highly relevant items (positives) from both irrelevant ones (easy negatives) and from somewhat related yet not competitive ones (hard negatives). Also, we have little control in the diversity and fairness of the retrieval results because of the ``greedy'' nature of nearest vector search. These issues compromise the performance of EBR methods in large-scale industrial scenarios. This paper introduces a simple and proven-in-production solution to overcome these issues. The proposed solution takes a divide-and-conquer approach: the whole set of candidate items are divided into multiple clusters and we run EBR to retrieve relevant candidates from each cluster in parallel; top candidates from each cluster are then combined by some controllable merging strategies. This approach allows our EBR models to only concentrate on discriminating positives from mostly hard negatives. It also enables further improvement from a multi-tasking learning (MTL) perspective: retrieval problems within each cluster can be regarded as individual tasks; inspired by recent successes in prompting and prefix-tuning, we propose an efficient task adaption technique further boosting the retrieval performance within each cluster with negligible overheads.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
