虚拟淘宝：虚拟化现实网络零售环境的强化学习
Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
摘要：
在物理世界任务中应用强化学习极具挑战性。在物理环境中，如当前强化学习方法所要求的，对大量试验进行采样通常是不可行的。本文报告了我们的项目使用强化学习更好的商品搜索在淘宝，最大的在线零售平台之一，同时具有高采样成本的物理环境。我们提出了一种方法，而不是直接在淘宝上训练强化学习：首先，我们建立虚拟淘宝，通过建议的GAN-SD（GAN for Simulating Distributions）和MAIL（multi-agent adversarial模仿学习）从历史客户行为数据中学习的模拟器，然后，我们训练虚拟淘宝策略，没有物理成本，其中ANC（Action Norm Constraint）策略被提出来减少过拟合。在实验中，虚拟淘宝是从数以亿计的客户记录中训练出来的，并将其属性与真实环境进行了比较。结果表明，虚拟淘宝忠实地恢复了现实环境的重要属性。我们还表明，在虚拟淘宝训练的政策可以有显着优越的在线性能比传统的监督方法。我们希望我们的工作可以为强化学习在复杂物理环境中的应用提供一些启示。
英文摘要：
Applying reinforcement learning in physical-world tasks is extremely challenging. It is commonly infeasible to sample a large number of trials, as required by current reinforcement learning methods, in a physical environment. This paper reports our project on using reinforcement learning for better commodity search in Taobao, one of the largest online retail platforms and meanwhile a physical environment with a high sampling cost. Instead of training reinforcement learning in Taobao directly, we present our approach: first we build Virtual Taobao, a simulator learned from historical customer behavior data through the proposed GAN-SD (GAN for Simulating Distributions) and MAIL (multi-agent adversarial imitation learning), and then we train policies in Virtual Taobao with no physical costs in which ANC (Action Norm Constraint) strategy is proposed to reduce over-fitting. In experiments, Virtual Taobao is trained from hundreds of millions of customers' records, and its properties are compared with the real environment. The results disclose that Virtual Taobao faithfully recovers important properties of the real environment. We also show that the policies trained in Virtual Taobao can have significantly superior online performance to the traditional supervised approaches. We hope our work could shed some light on reinforcement learning applications in complex physical environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
