电子商务平台通常会在Feed中显示广告和有机项目的混合列表。一个关键的问题是如何分配提要中有限的位置，以最大限度地提高整体收入并改善用户体验，这需要一个好的用户偏好模型。代替对单个项目对用户行为的影响进行建模，布置信号对项目的布置的影响进行建模，并且可以导致更好的分配策略。然而，大多数以前的策略未能模拟这样的信号，因此导致次优性能。此外，广告曝光率（PAE）是广告分配的重要指标。过高的PAE会损害用户体验，而过低的PAE会降低平台收入。因此，如何将PAE约束在一定范围内，同时保持个性化推荐在PAE约束下是一个挑战。本文提出了交叉深度Q网络（Cross Deep Q Network，Cross DQN），通过交叉不同项目的嵌入，对交叉序列进行多通道注意力建模，提取关键排列信号。此外，我们提出了一个辅助损失的批量级约束PAE，以解决上述挑战。我们的模型在离线实验中比最先进的基线产生更高的收入和更好的用户体验。此外，我们的模型在在线A/B测试中表现出了显著的改进，并已在美团饲料上全面部署，为超过3亿客户提供服务。
英文摘要：
E-commerce platforms usually display a mixed list of ads and organic items in feed. One key problem is to allocate the limited slots in the feed to maximize the overall revenue as well as improve user experience, which requires a good model for user preference. Instead of modeling the influence of individual items on user behaviors, the arrangement signal models the influence of the arrangement of items and may lead to a better allocation strategy. However, most of previous strategies fail to model such a signal and therefore result in suboptimal performance. In addition, the percentage of ads exposed (PAE) is an important indicator in ads allocation. Excessive PAE hurts user experience while too low PAE reduces platform revenue. Therefore, how to constrain the PAE within a certain range while keeping personalized recommendation under the PAE constraint is a challenge. In this paper, we propose Cross Deep Q Network (Cross DQN) to extract the crucial arrangement signal by crossing the embeddings of different items and modeling the crossed sequence by multi-channel attention. Besides, we propose an auxiliary loss for batch-level constraint on PAE to tackle the above-mentioned challenge. Our model results in higher revenue and better user experience than state-of-the-art baselines in offline experiments. Moreover, our model demonstrates a significant improvement in the online A/B test and has been fully deployed on Meituan feed to serve more than 300 millions of customers.
深度Q网络（DQN）是一个非常成功的算法，但强化学习的固有问题，即开发-探索平衡，仍然存在。在这项工作中，我们引入熵正则化到DQN，并提出了SQN。我们发现，软Q学习的备份方程可以享受正确的反馈，如果我们把软备份看作是Q形式的策略改进，而不是策略评估。我们证明了带纠正反馈的软Q学习（SQL-CF）是SQL的on-plicy性质以及SQL和软策略梯度（SPG）的等价性的基础。有了这些见解，我们提出了一个策略版本的深度Q学习算法，即Q On-Policy（QOP）。我们在一个名为Google Research Football（GRF）的自我游戏环境中进行了QOP实验。QOP算法在训练GRF代理时表现出很好的稳定性和效率。
An Integrated Approach to Neural Architecture Search for Deep Q-Networks
Iman Rahmani Intelligent Control Systems Institute (ICSI) Department of Aerospace Engineering K. N. Toosi University of Technology Tehran, Iran（智能控制系统研究所（ICSI）航空航天工程系，K.N.托菲大学技术学院，泰赫兰，伊朗）；Saman Yazdannik Machine Learning Engineer Isfahan Chamber of Commerce, Industries, Mines and Agriculture (ECCIMA) Isfahan, Iran（机器学习工程师，以沙万商会、工业、采矿和农业（ECCIMA））；Morteza Tayefi Department of Aerospace Engineering K. N. Toosi University of Technology Tehran, Iran（航空航天工程系，K.N.托菲大学技术学院，泰赫兰，伊朗）；Jafar Roshanian Intelligent Control Systems Institute (ICSI) Department of Aerospace Engineering K. N. Toosi University of Technology Tehran, Iran（智能控制系统研究所（ICSI）航空航天工程系，K.N.托菲大学技术学院，泰赫兰，伊朗）
