arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.08403cs.AI
收藏

Owen-Shapley策略优化:一种为生成搜索语言模型设计的原理性强化学习算法

Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs

Amazon Science(亚马逊科学);Situated Grounding and Natural Language (SIGNAL) Lab, Colorado State University(情境 grounding 和自然语言(SIGNAL)实验室,科罗拉多州立大学)

Abhijnan Nath, Alireza Bagheri Garakani, Tianchen Zhou, Fan Yang, Yan Gao, Nikhil Krishnaswamy

AI总结:

本文提出Owen-Shapley策略优化算法,通过基于令牌边际贡献的序列优势再分配,解决生成搜索语言模型中因稀疏序列奖励导致的信用分配问题,实验表明其在Amazon ESCI和H&M Fashion数据集上表现优异。

摘要:

大型语言模型越来越多地通过强化学习进行训练,以实现个性化推荐任务,但标准方法如GRPO依赖于稀疏的序列级奖励。这些奖励模糊了哪些令牌实际上对高质量输出有贡献,导致信用分配问题。当模型必须从未指定的语言中推断潜在用户意图时,这一问题尤为突出,这在预训练期间很少见,但在部署时却很常见。我们引入Owen-Shapley策略优化(OSPO),一种框架,通过基于令牌边际贡献的序列优势再分配,将任务反馈转化为基于潜力的奖励塑造。OSPO通过Shapley-Owen归因将段级信用分配,同时保持最优策略,而无需参数化的价值模型。通过形成语义上连贯的单元(例如描述产品属性的短语或捕捉偏好的句子),OSPO确定哪些响应部分驱动性能。在Amazon ESCI和H&M Fashion数据集上的实验,包括受控生成任务,显示OSPO在基线模型上表现一致,并在训练期间未见过的分布外检索器上表现出显著的测试时间鲁棒性。

英文摘要:

Large language models are increasingly trained via reinforcement learning for personalized recommendation tasks, but standard methods like GRPO rely on sparse, sequence-level rewards. These obscure which tokens actually contribute to high-quality outputs, creating a credit assignment gap. This gap is especially problematic when models must infer latent user intent from under-specified language without ground truth labels, which is a reasoning pattern rarely seen during pretraining but commonly required in deployment. We introduce Owen-Shapley Policy Optimization (OSPO), a framework that redistributes sequence-level advantages based on tokens' marginal contributions to outcomes. OSPO transforms task feedback into potential-based reward shaping via Shapley-Owen attributions to assign segment-level credit while preserving the optimal policy, all without parametric value models. By forming coalitions of semantically coherent units (e.g., phrases describing product attributes or sentences capturing preferences), OSPO identifies which response parts drive performance. Experiments on Amazon ESCI and H&M Fashion datasets including controlled generation tasks show consistent gains over baselines and notable test-time robustness to out-of-distribution retrievers unseen during training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑