Generator and Critic：一种基于深度强化学习的电子商业Slate重排序方法
Generator and Critic: A Deep Reinforcement Learning Approach for Slate Re-ranking in E-commerce
摘要：
与逐点排序相比，石板重排序问题考虑了项目之间的相互影响，以提高电子商业中的用户满意度。以前的作品要么直接排名项目的端到端的模型，或排名项目的评分功能，权衡点的分数和项目之间的差异。然而，有两个主要的现有挑战没有得到很好的研究：（1）由于一个石板中的项目之间的复杂的相互影响，石板的评估是困难的;（2）即使给出最优评估，搜索最优石板也是具有挑战性的，因为动作空间是指数级的。在本文中，我们提出了一种新的生成器和评论家石板重新排名的方法，其中的评论家评估石板和生成器排名的项目通过强化学习的方法。我们提出了一个完整的石板批评（FSC）模型，认为真正的印象深刻的项目，并避免了现有模型的印象偏见。对于Generator，为了解决大动作空间的问题，我们提出了一种新的探索强化学习算法，称为PPO-Exploration。实验结果表明，FSC模型的性能明显优于现有的评价方法，PPO-Exploration算法的性能明显优于现有的强化学习方法。Generator和Critic方法在世界上最大的电子商业网站之一的现场实验中提高了石板效率（4%的gmv和5%的订单数）和多样性。
英文摘要：
The slate re-ranking problem considers the mutual influences between items to improve user satisfaction in e-commerce, compared with the point-wise ranking. Previous works either directly rank items by an end to end model, or rank items by a score function that trades-off the point-wise score and the diversity between items. However, there are two main existing challenges that are not well studied: (1) the evaluation of the slate is hard due to the complex mutual influences between items of one slate; (2) even given the optimal evaluation, searching the optimal slate is challenging as the action space is exponentially large. In this paper, we present a novel Generator and Critic slate re-ranking approach, where the Critic evaluates the slate and the Generator ranks the items by the reinforcement learning approach. We propose a Full Slate Critic (FSC) model that considers the real impressed items and avoids the impressed bias of existing models. For the Generator, to tackle the problem of large action space, we propose a new exploration reinforcement learning algorithm, called PPO-Exploration. Experimental results show that the FSC model significantly outperforms the state of the art slate evaluation methods, and the PPO-Exploration algorithm outperforms the existing reinforcement learning methods substantially. The Generator and Critic approach improves both the slate efficiency(4% gmv and 5% number of orders) and diversity in live experiments on one of the largest e-commerce websites in the world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
