arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.20150cs.IR
收藏

Rank-GRPO: 基于强化学习训练基于大语言模型的对话推荐系统

Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning

University of Virginia(弗吉尼亚大学);Netflix Research(Netflix研究部);Cornell University(康奈尔大学)

Yaochen Zhu, Harald Steck, Dawen Liang, Yinhan He, Vito Ostuni, Jundong Li, Nathan Kallus

AI总结:

Rank-GRPO通过两阶段框架提升LLM对话推荐系统的训练效果,解决排名质量下降问题。

摘要:

大语言模型(LLMs)通过使用户能够通过对话表达偏好并接收推荐,正在重塑推荐系统范式。然而,将LLMs对齐到推荐任务仍具挑战性:预训练的LLMs通常生成超出目录的商品、违反所需输出格式,并且其排名质量在生成列表末尾急剧下降。为此,我们提出了ConvRec-R1,一个用于端到端训练基于LLM的对话推荐系统的两阶段框架。在第一阶段,我们构建了一个行为克隆数据集,通过Remap-Reflect-Adjust流程,从强大的黑盒LLMs生成高质量的目录导向演示,以预热RL训练。在第二阶段,我们提出了Rank-GRPO,一种专门针对具有排名式输出任务的组相对策略优化(GRPO)的扩展。Rank-GRPO将推荐列表中的每个排名视为单位,而不是令牌(过于精细)或序列(过于粗略),重新定义奖励以消除非因果信用分配,并引入基于排名级令牌概率几何均值的排名级重要性比值以稳定策略更新。在公共Reddit-v2数据集上的实验表明,ConvRec-R1比GRPO式基线收敛更快,Recall和NDCG更高。代码和数据集已发布在https://github.com/yaochenzhu/Rank-GRPO。

英文摘要:

Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑