arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02779cs.IRcs.AIcs.LG
收藏

PrefRec:基于人类偏好以增强长期用户参与度的推荐系统

PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

Nanyang Technological University(南洋理工大学);Kuaishou Technology(快手科技)

Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Kun Gai, Bo An

AI总结:

针对推荐系统中长期参与度奖励难以设计的问题,本文提出PrefRec,利用用户历史行为偏好端到端训练奖励函数并优化RL推荐策略,在多项任务上显著超越现有方法。

摘要:

当前推荐系统的进展在优化即时参与度方面取得了显著成功。然而,长期用户参与度作为一种更理想的性能指标,仍然难以提升。与此同时,近期的强化学习(RL)算法已在多种长期目标优化任务中展现出有效性。因此,RL被广泛认为是在推荐中优化长期用户参与度的一种有前景的框架。尽管前景良好,RL的应用在很大程度上依赖于设计良好的奖励,但设计与长期用户参与度相关的奖励相当困难。为缓解这一问题,我们提出了一种新颖范式,即基于人类偏好的推荐系统(或称基于偏好的推荐系统),它使RL推荐系统能够从关于用户历史行为的偏好中学习,而不是依赖显式定义的奖励。这类偏好无需任何专业知识,因此可通过众包等技术轻松获取。借助PrefRec,我们可以充分利用RL在优化长期目标方面的优势,同时避免复杂的奖励工程。PrefRec使用这些偏好以端到端方式自动训练奖励函数。随后,该奖励函数被用于生成学习信号,以训练推荐策略。此外,我们为PrefRec设计了一种有效的优化方法,该方法使用额外的价值函数、期望分位数回归和奖励模型预训练来提升性能。我们在多种长期用户参与度优化任务上进行了实验。结果表明,PrefRec在所有任务上均显著优于以往的最先进方法。

英文摘要:

Current advances in recommender systems have been remarkably successful in optimizing immediate engagement. However, long-term user engagement, a more desirable performance metric, remains difficult to improve. Meanwhile, recent reinforcement learning (RL) algorithms have shown their effectiveness in a variety of long-term goal optimization tasks. For this reason, RL is widely considered as a promising framework for optimizing long-term user engagement in recommendation. Though promising, the application of RL heavily relies on well-designed rewards, but designing rewards related to long-term user engagement is quite difficult. To mitigate the problem, we propose a novel paradigm, recommender systems with human preferences (or Preference-based Recommender systems), which allows RL recommender systems to learn from preferences about users historical behaviors rather than explicitly defined rewards. Such preferences are easily accessible through techniques such as crowdsourcing, as they do not require any expert knowledge. With PrefRec, we can fully exploit the advantages of RL in optimizing long-term goals, while avoiding complex reward engineering. PrefRec uses the preferences to automatically train a reward function in an end-to-end manner. The reward function is then used to generate learning signals to train the recommendation policy. Furthermore, we design an effective optimization method for PrefRec, which uses an additional value function, expectile regression and reward model pre-training to improve the performance. We conduct experiments on a variety of long-term user engagement optimization tasks. The results show that PrefRec significantly outperforms previous state-of-the-art methods in all the tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑