个性化电子商务策略的在线控制实验：设计，挑战和陷阱
Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls
摘要：
在线受控实验是衡量数字化企业产品变化的因果影响的主要工具。数字产品和服务以个性化的方式与客户互动越来越普遍。使用在线控制实验来优化个性化交互策略是具有挑战性的，因为违反了统计上等同的用户组的通常假设。此外，用户有资格根据动态，随机属性的策略引入的挑战。传统的A/B测试可以通过将用户预先分配到控制组和暴露组来挽救统计等效性，但这会稀释实验指标并降低测试功效。我们提出了一个堆栈增量测试框架，解决了个性化用户策略的在线实验运行的问题。我们推导出的界限表明，我们的框架是优于最好的简单的A/B测试有足够的用户，这个条件很容易满足大规模的在线实验。此外，我们还提供了一个测试功率计算器，并描述了一些陷阱和从我们使用它的经验中吸取的教训。
英文摘要：
Online controlled experiments are the primary tool for measuring the causal impact of product changes in digital businesses. It is increasingly common for digital products and services to interact with customers in a personalised way. Using online controlled experiments to optimise personalised interaction strategies is challenging because the usual assumption of statistically equivalent user groups is violated. Additionally, challenges are introduced by users qualifying for strategies based on dynamic, stochastic attributes. Traditional A/B tests can salvage statistical equivalence by pre-allocating users to control and exposed groups, but this dilutes the experimental metrics and reduces the test power. We present a stacked incrementality test framework that addresses problems with running online experiments for personalised user strategies. We derive bounds that show that our framework is superior to the best simple A/B test given enough users and that this condition is easily met for large scale online experiments. In addition, we provide a test power calculator and describe a selection of pitfalls and lessons learnt from our experience using it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
