arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09215cs.IRcs.AI
收藏

关于推荐系统中Softmax直接偏好优化的研究

On Softmax Direct Preference Optimization for Recommendation

National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Hokkaido University(北海道大学)

Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, Tat-Seng Chua

AI总结:

提出Softmax-DPO方法,将排序信息注入语言模型推荐系统,通过多负样本和部分排序损失提升个性化推荐性能,并在三个数据集上验证其有效性。

摘要:

推荐系统旨在基于用户偏好数据预测个性化排序。随着语言模型(LMs)的兴起,基于LM的推荐系统因其广泛的世界知识和强大的推理能力而受到广泛探索。大多数基于LM的推荐系统将历史交互转换为语言提示,与作为目标响应的正样本配对,并使用语言建模损失对LM进行微调。然而,当前的目标未能充分利用偏好数据,且并非针对个性化排序任务进行优化,这阻碍了基于LM的推荐系统的性能。受当前直接偏好优化(DPO)在人类偏好对齐方面的进展以及softmax损失在推荐中的成功启发,我们提出了Softmax-DPO(S-DPO),将排序信息注入LM,帮助基于LM的推荐系统区分偏好项与负样本,而不仅仅关注正样本。具体而言,我们在用户偏好数据中纳入多个负样本,并设计了一种针对基于LM的推荐系统量身定制的DPO损失替代版本,该版本从传统的全排序Plackett-Luce(PL)模型扩展到部分排序,并与softmax采样策略相关联。理论上,我们将S-DPO与负采样上的softmax损失联系起来,发现其具有挖掘难负样本的内在优势,这确保了其在推荐任务中的卓越能力。实证上,在三个真实世界数据集上进行的大量实验表明,S-DPO在有效建模用户偏好和进一步提升推荐性能方面具有优越性,同时为偏好项提供更好的奖励。我们的代码可在https://github.com/chenyuxin1999/S-DPO获取。

英文摘要:

Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (S-DPO) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑