基于预算分割设计的可信在线市场实验
Trustworthy Online Marketplace Experimentation with Budget-split Design
摘要:
在线实验,也称为A/B测试,是衡量产品影响力和科技行业商业决策的黄金标准。然而,实验的有效性和实用性取决于无偏性和足够的力度。在双边在线市场中,这两个要求都受到质疑。伯努利随机试验是有偏差的,因为治疗单位通过市场竞争干扰控制单位,违反了“稳定单位治疗值假设”(SUTVA)。由于市场两边的样本量不同,至少市场一边的实验效力往往不足。尽管在线市场对在线经济的重要性以及实验在产品改进中的关键作用,但缺乏有效和实用的解决方案来解决市场实验中的偏见和低功耗问题。我们的论文填补了这一空白,提出了一个试验设计,是公正的,在任何市场上,买家有一个明确的预算,这可能是有限的或无限的。我们表明,它是更强大的比所有其他文献中的无偏设计。然后,我们提供了可推广的系统架构部署这种设计的在线市场。最后,我们证实了我们的研究结果与实证性能的实验运行在两个现实世界的在线市场。
英文摘要:
Online experimentation, also known as A/B testing, is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. The Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition and violate the "stable unit treatment value assumption"(SUTVA). The experimental power on at least one side of the market is often insufficient because of disparate sample sizes on the two sides. Despite the important of online marketplaces to the online economy and the crucial role experimentation plays in product improvement, there lacks an effective and practical solution to the bias and low power problems in marketplace experimentation. Our paper fills this gap by proposing an experimental design that is unbiased in any marketplace where buyers have a defined budget, which could be finite or infinite. We show that it is more powerful than all other unbiased designs in literature. We then provide generalizable system architecture for deploying this design to online marketplaces. Finally, we confirm our findings with empirical performance from experiments run in two real-world online marketplaces.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。