arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.09141cs.IRcs.AIcs.LG
收藏

强化学习在冷启动推荐中的生命周期价值优化

Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendation

DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Center for Data Science, AAIS, Peking University(北京大学人工智能研究院数据科学中心)

Luo Ji, Qin Qi, Bingqing Han, Hongxia Yang

摘要:

推荐系统在现代电子商务平台中起着至关重要的作用。由于用户和项目之间缺乏历史交互,冷启动推荐是一个具有挑战性的问题。为了缓解冷启动问题,大多数现有方法引入内容和上下文信息作为辅助信息。然而,这些方法假设推荐的项目随着时间的推移表现稳定,而在典型的电子商务场景中,项目通常在其整个生命周期中具有非常不同的性能。在这种情况下,从项目的角度考虑长期回报将是有益的,这在传统方法中通常被忽略。强化学习(Reinforcement learning,RL)很好地解决了这一长期优化问题,即推荐器可以识别出高潜力的项目,主动分配更多的用户印象以促进其增长,从而提高多期累积收益。受此思想的启发,我们将该过程建模为部分可观察和可控制的马尔可夫决策过程(POC-MDP),并提出了一个行动者-评论家强化学习框架(RL-LTV),将项目生命周期值(LTV)纳入推荐。在RL-LTV模型中,评价者通过研究物品的历史轨迹来预测新鲜物品的未来LTV,而行动者则提出一种基于分数的策略来最大化未来LTV的期望。然后,由演员建议的分数与双等级框架中的经典排名分数相结合,因此建议与LTV考虑相平衡。我们的方法优于强实时基线,在最大的电子商务平台之一上,冷启动项目的IPV和GMV相对提高了8.67%和18.03%。

英文摘要:

Recommender system plays a crucial role in modern E-commerce platform. Due to the lack of historical interactions between users and items, cold-start recommendation is a challenging problem. In order to alleviate the cold-start issue, most existing methods introduce content and contextual information as the auxiliary information. Nevertheless, these methods assume the recommended items behave steadily over time, while in a typical E-commerce scenario, items generally have very different performances throughout their life period. In such a situation, it would be beneficial to consider the long-term return from the item perspective, which is usually ignored in conventional methods. Reinforcement learning (RL) naturally fits such a long-term optimization problem, in which the recommender could identify high potential items, proactively allocate more user impressions to boost their growth, therefore improve the multi-period cumulative gains. Inspired by this idea, we model the process as a Partially Observable and Controllable Markov Decision Process (POC-MDP), and propose an actor-critic RL framework (RL-LTV) to incorporate the item lifetime values (LTV) into the recommendation. In RL-LTV, the critic studies historical trajectories of items and predict the future LTV of fresh item, while the actor suggests a score-based policy which maximizes the future LTV expectation. Scores suggested by the actor are then combined with classical ranking scores in a dual-rank framework, therefore the recommendation is balanced with the LTV consideration. Our method outperforms the strong live baseline with a relative improvement of 8.67% and 18.03% on IPV and GMV of cold-start items, on one of the largest E-commerce platform.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑