arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.12995cs.LG
收藏

F-GRPO:基于统一候选生成与排序的因子化组相对策略优化

F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking

UC San Diego(南加州大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Adobe Research(Adobe研究院)

Rohan Surana, Gagan Mundada, Junda Wu, Xintong Li, Yizhu Jiao, Bowen Jin, Sizhe Zhou, Tong Yu, Ritwik Sinha, Jiawei Han, Jingbo Shang, Julian McAuley

AI总结:

本文提出F-GRPO,通过因子化组相对策略优化统一完成候选生成与排序,解决传统方法中因反馈滞后导致的优化不稳定问题,提升推荐系统性能。

摘要:

传统检索流程通过候选检索和重排序阶段优化效用,其中排序操作在预定义候选集上进行。大语言模型(LLMs)扩展了这一过程:给定候选池,LLM可以在单次自回归传递中生成子集并对其进行排序。然而,这种灵活性引入了新的优化挑战:模型必须在生成完整排序列表后才能获得效用反馈,无法区分低效结果是由于未能生成相关子集还是未能正确排序该子集。这种信用分配差距使端到端优化不稳定且样本效率低下。现有系统通常通过将候选生成与排序分离来解决这一问题。然而,这种解耦仍与下游效用不匹配,因为排序受限于所接收的候选集。为此,我们提出一个统一框架,在单次自回归滚动中同时执行两者,并通过因子化组相对策略优化(F-GRPO)进行端到端优化。我们的框架将策略分解为候选生成和排序,同时共享一个LLM主干,并使用顺序不变覆盖奖励和位置感知效用奖励进行联合训练。为解决由此产生的阶段特定信用分配问题,我们在两阶段序列级目标中使用分开的组相对优势。在序列推荐和多跳问答基准上,F-GRPO在GRPO和解耦基线之上提升了前排性能,优于监督替代方案,并与强零样本重排序器竞争,且在推理时无需架构更改。

英文摘要:

Traditional retrieval pipelines optimize utility through stages of candidate retrieval and reranking, where ranking operates over a predefined candidate set. Large Language Models (LLMs) broaden this into a generative process: given a candidate pool, an LLM can generate a subset and order it within a single autoregressive pass. However, this flexibility introduces a new optimization challenge: the model must search a combinatorial output space while receiving utility feedback only after the full ranked list is generated. Because this feedback is defined over the completed sequence, it cannot distinguish whether a poor result arises from failing to generate a relevant subset or from failing to rank that subset correctly. This credit assignment gap makes end-to-end optimization unstable and sample-inefficient. Existing systems often address this by separating candidate generation from ranking. However, such decoupling remains misaligned with downstream utility because ranking is limited by the candidate set it receives. To bridge this gap, we propose a unified framework that performs both within a single autoregressive rollout and optimizes them end-to-end via factorized group-relative policy optimization (F-GRPO). Our framework factorizes the policy into candidate generation and ranking while sharing a single LLM backbone, and jointly trains them with an order-invariant coverage reward and a position-aware utility reward. To address the resulting phase-specific credit assignment problem, we use separate group-relative advantages for generation and ranking within a two-phase sequence-level objective. Across sequential recommendation and multi-hop question answering benchmarks, F-GRPO improves top-ranked performance over GRPO and decoupled baselines, outperforms supervised alternatives, and remains competitive with strong zero-shot rerankers, with no architectural changes at inference time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑