arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26385cs.IRcs.AIstat.ML
收藏

两阶段排序中早期检索的信用分配策略梯度

Credit-assigned Policy Gradient for Early Stage Retrieval in Two-stage Ranking

Computer Science Department, Cornell University, Ithaca, NY, USA(康奈尔大学计算机科学系);Central Applied Science, Meta, Menlo Park, CA, USA(Meta中央应用科学)

Haruka Kiyohara, Mihaela Curmei, Ariel Evnine, Shankar Kalyanaraman, Israel Nir, Ana-Roxana Pop, Nitzan Razin, Sarah Dean, Thorsten Joachims, Udi Weinsberg

AI总结:

针对两阶段排序中早期排序器(ESR)端到端训练难的问题,提出信用分配策略梯度(CA-PG),通过对目标项被选中的概率求梯度来降低方差,提升训练稳定性和收敛速度。

摘要:

大规模搜索、推荐和检索增强生成(RAG)系统通常采用两阶段架构:早期排序器(ESR)生成候选集,随后由后期排序器(LSR)重新排序。虽然有许多强化学习(RL)方法用于训练LSR,但ESR的端到端训练被证明具有挑战性。特别是,朴素应用“普通”策略梯度(V-PG)对于实际使用的候选集大小不可扩展,因为方差爆炸。该问题源于V-PG将梯度传播到候选集的联合概率,忽略了候选集中每个特定项对奖励的贡献。为缓解此问题,我们提出了一种新颖的“信用分配”策略梯度(CA-PG),它计算相对于目标项在任何候选集中被选中的概率的梯度,即边际化所有包含它的候选集。我们的理论分析表明,CA-PG通过边际化候选集的具体组成显著降低了V-PG的方差,同时保留了在合理对齐的LSR策略下学习正确排序项的能力。在合成和真实数据上的实验表明,CA-PG提高了使用经典Plackett-Luce模型的ESR的收敛速度和训练稳定性,特别是在候选集大小较大时。

英文摘要:

Large-scale search, recommendation, and retrieval-augmented generation (RAG) systems typically employ a two-stage architecture: an early-stage ranker (ESR) generates a candidate set, which is subsequently re-ranked by a late-stage ranker (LSR). While there are many reinforcement learning (RL) methods for training the LSR, end-to-end training of the ESR has proven challenging. In particular, naive application of "vanilla" policy gradient (V-PG) is not scalable for candidate-set sizes relevant for practical use due to exploding variance. This issue arises because V-PG propagates the gradient to the joint probability of the candidate sets, ignoring the contribution of each specific item in the candidate set to the reward. To mitigate this issue, we propose a novel "credit-assigned" policy gradient (CA-PG), which computes gradients with respect to the probability that the target item is chosen in any candidate set, i.e. marginalizing over all candidate sets that contain it. Our theoretical analysis reveals that CA-PG significantly reduces the variance of V-PG by marginalizing over the specific composition of the candidate set, while preserving the ability to learn the correct ranking of items under a reasonably aligned LSR policy. Experiments on both synthetic and real-world data demonstrate that CA-PG improves the convergence speed and training stability for ESRs utilizing the canonical Plackett-Luce model, especially when the candidate-set size is large.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑